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EXECUTIVE SUMMARY 



EXECUTIVE SUMMARY 



Connie A. Bridge, Peter N. Winograd and Joseph M. Petrosko 
University of Kentucky and University of Louisville 



Background for this Review of Research. 

In June of 1990, the Kentucky legislature passed the most sweeping educational 
reform act in the history of the United States, the Kentucky Education Reform Act (KERA). 
In response to the Kentucky Supreme Court ruling that the schools were inequitable and 
inefficient, the legislature totally revamped Kentucky’s educational system in the areas of 
finance, governance and curriculum in an attempt to provide equal educational opportunities 
for all of Kentucky’s children regardless of the property wealth of the district in which they 
lived. 



The Kentucky Education Reform Act is unique because unlike the reforms in many 
other states, it was not limited to setting higher educational standards and creating new 
organizational structures, introducing new statewide assessment accompanied by rewards and 
sanctions. In addition, KERA created additional support systems for teachers, families, and 
students in the form of increased funding for staff development to help teachers learn to 
implement the new mandates, a preschool program for economically at-risk four-year-olds and 
children with disabilities, an extended school services program for students who need more 
time to learn before or after school or in the summer, and family resource youth service 
centers to put students and their families in touch with needed health and social services. 

As we take stock of where we are now five years into the reform, we must not forget 
where we came from and the compelling reasons for the initial passage of the Kentucky 
Education Reform Act. In 1989 when the Supreme Court handed down its decision, 

Kentucky was notorious for its traditional failure to support education, for its high dropout 
rate and its high illiteracy rate, and for the limited supply of qualified workers available to 
attract business and industry. 

Now Kentucky is recognized nationally for its ambitious and comprehensive attempt to 
improve its educational system. Even as we suffer from the pain and discomfort of change 
and struggle to solve some of the inevitable problems associated with implementation, we 
must not delude ourselves into thinking that the pre-KERA days were the "good old days" in 
Kentucky education. Our own research tell us they were not. 



Purpose of this Review of Research. 

The purpose of this review of research is to identify, review, and summarize studies 
that address the implementation of KERA and that focus on the effects of reform on students, 
teachers, and other stakeholders. It is important to stress that this is a summary of the 
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research that has been conducted on KERA, not a comprehensive status report on the 
implementation of the KERA mandates. Other agencies, including the Kentucky Department 
of Education and the Office of Accountability, publish annual reviews that focus on KERA’s 
programs, practices, and activities. This report focuses on research about KERA because it 
provides us with a systematic opportunity to reflect on reform and evaluate the massive 
changes taking place in Kentucky. 

In this executive summary, you will find introductory overviews of the research in the 
areas of finance, governance, and curriculum. In addition, we will focus on three areas that 
are receiving a great deal of current attention: assessment and accountability, the primary 
program, and the need for professional development. We end with synopses of 15 major 
aspects of KERA. These synopses summarize what we know and do not know about the 
nature and extent of implementation, the effects on major stakeholders, and the attitudes and 
perceptions of these stakeholders. The major aspects of KERA will be discussed in the 
following order: 

FINANCE 

School Finance Reform 

GOVERNANCE 

School-Based Decision Making 

Reorganization of the Kentucky Department of Education 
Education Professional Standards Board 

CURRICULUM 

Assessment and Accountability 

The KERA Preschool Programs for At-Risk 4-Year-Old Children and 3- and 4- Year-Old 
Children with Disabilities 
Primary Program 
High School Restructuring 
Extended School Services 
Education Technology 
Professional Development 
Curriculum Framework 
Multicultural Education 

KERA Family Resource and Youth Services Centers 

PERCEPTIONS AND BELIEFS ABOUT KERA 

Perceptions and Beliefs about Education Reform in Kentucky 
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What Does Research Tell Us About Education Reform in Kentucky? 

The research indicates that during the past five years, Kentucky has made great strides 
in the implementation of this comprehensive reform effort. However, there are many areas of 
concern and many problems to address before the vision of KERA is fully realized. The 
discussion that follows presents evidence of both successes and problems in the areas of 
finance, governance, and curriculum. 



Finance 



In the area of financial inequity which served as the initial impetus for KERA, 
Kentucky has moved toward the equalization of educational funding throughout the state, 
having reduced by half the gap in spending between the wealthiest and poorest districts. 
Furthermore, overall spending on education has moved Kentucky into 32nd place in per pupil 
expenditures for education. Nevertheless, the level of per pupil support for education is still 
significantly below the national average. 

Teachers’ salaries have increased by 19.8% on the average (6.3% when corrected for 
inflation) with the highest percentage increases in property-poor districts. Still average 
teacher salaries lag behind the national average. Importantly, 72.4% of the new funds made 
available under KERA have been allocated to instruction with only 4.2% to administration. 
Little evidence is yet available regarding whether or not the increased spending on education 
in general and on instruction specifically has translated into higher student achievement. 



Governance 



In the area of educational governance, progress has been made in transferring control 
of schools to the individual school level through school-based decision making councils. By 
October 1995 approximately 925 of Kentucky’s 1,365 schools had established school-based 
decision making (SBDM) councils. The remaining 450 schools are expected to establish 
councils by July 1996 unless they have been exempted by virtue of having reached their 
threshold goals on the Kentucky assessment system. 

Several issues remain unresolved in the area of governance: the relationship between 
the school councils and the district school boards has not been clearly defined and 
disagreements over areas of authority have occurred; councils have been slow to assume 
authority in many of the eight areas in which power has been delegated; parent involvement 
in councils and council committees remains low and several groups are pushing for increasing 
the number of parents on the councils to achieve equal representation among teachers and 
parents; in some schools principals’ leadership styles tend to limit participation of teacher and 
parent council members; and time demands of council participation present problems for 
members. 
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More research is needed on ways to increase parent participation, to remove barriers to 
the establishment of councils in schools which have delayed the formation of a council. In 
spite of the time demands and other problems, participants in SBDM report favorable 
attitudes toward the councils as a means of school governance. 



Curriculum 



Research on the statewide Preschool Program for at-risk low income children and 
children with disabilities indicates that program quality has improved since the initit ’ year of 
implementation and that this early intervention program has helped close the achievement gap 
between low-income children who participated in the program and their economically more 
fortunate peers. 

Studies of the Extended School Services Program show that large number of students 
have raised their grades, have passed courses, and/or have promoted or graduated as a result 
of the help they received in the ESS programs. All stakeholders — educators, parents, and 
students — agree that the program has had a positive effect on the academic success of the 
students who participated in the program. However, there is little information about what 
types of students are receiving the benefits from this program. 

Research on the Family Resource Youth Service Centers (FRYSC’s) also reveals 
positive evaluations by the major stakeholders affected by the centers. The provision of 
health and social services has reached its intended audience of needy at-risk children and 
families and is judged to be contributing to the well-being of these children and families. 

Some initial research indicates positive effects of services on students’ achievement in school 
as judged by the teachers of these students. 

The installation of the Kentucky Educational Technology System has progressed 
significantly during the past two years, propelling Kentucky to the rank of seventh in the 
nation in terms of numbers of computers per student. The 1994-1995 years were utilized 
building the capacity of schools to network. All stakeholders - students, teachers, parents, 
and the public - support the use of increased technology in schools. However, teachers still 
make limited use of computers in instruction, citing the need for more staff development and 
time to learn appropriate applications to instructional and administrative tasks. 

This brief overview of research related to finance, governance, and curriculum 
indicates that many aspects of KERA are working well. It is important to note that initiatives 
that are working well, like the Preschool Program, the Extended School Services Program, the 
Family Resource Youth Services Centers, and the Kentucky Educational Technology System, 
tend to be "add-ons" and require the least amount of change on the part of the major 
stakeholders. These "add-ons" are less demanding on the teachers who must implement them 
and are less apt to run counter to established norms and practices. On the other hand, aspects 
of reform that have required fundamental changes in the way that teachers teach, students are 
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viewed, and administrators govern are the more controversial. We now turn our attention to 
three areas of KERA that are the subject of a great deal of current discussion and debate. 

Key Issues in KERA. 

The assessment and accountability and the primary program are two areas that were 
particularly controversial in 1995, live years into the implementation of KERA. The third 
area, the need for professional development, is not so much a controversial topic. Rather, we 
have come to appreciate how crucial systematic, long-term professional development is to the 
success of education reform. In the next section, we will examine each of these topics in 
more detail. 



Assessment and Accountability System 

Of all the KERA initiatives, the area of assessment and accountability was probably 
the most controversial in 1995. Educators, policy makers, and the general public have 
expressed concerns about the statewide assessment and accountability system. There were 
several reasons for this. The most obvious was that 1995 saw the end of the first 
"accountability cycle" mandated in KERA. Based on comparisons between data collected in 
1991-92 and the average of data collected in 1992-93 and 1993-94, schools were assigned 
into one of five categories: (a) eligible for monetary rewards, (b) successful, (c) not meeting 
threshold or improving, (d) in decline, and (e) in crisis. 

Although not every school in the state made achievement gains, the Kentucky 
Department of Education (KDE) touted the overall gains that were made, especially by 
schools with grade four or grade eight. Educators in the schools labeled "eligible for monetary 
rewards" received approximately 26 million dollars in bonus money in 1995. 

Because of changes in KERA in 1994, schools in the lowest category, "in crisis," did 
not suffer the extreme sanction of having all of its certified staff placed on probation, one of 
the penalties in the original version of the reform law. However, schools in this category, 
and all the other categories, except for "eligible for monetary rewards" and "successful," had 
to develop a school improvement plan. And many schools were assigned a "Kentucky 
distinguished educator" to advise school personnel on implementing such a plan. 

Other developments related to assessment were given a prominent place in the news 
media coverage of KERA. Two major evaluations of the assessment and accountability 
systems were published in 1995. Both contained criticisms of the current systems and 
recommendations for changes. The first evaluation to be released (dated January 1995) was 
sponsored by the Kentucky Institute for Education Research (KIER). In this study, a team of 
evaluators from Western Michigan University praised some aspects of the Kentucky 
assessment— for example the innovative approach taken in assessment, and the positive 
benefits of writing portfolios (the latter having led to increased student writing). But the 
evaluators had numerous criticisms and suggestions. For example, that multiple-choice items 
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be included in the assessment. (Multiple-choice items have been replaced by open-response 
items in the current versions of KIRIS.) In addition, the Western Michigan evaluators 
expressed concerns about the methods used to calculate the reliability of KIRIS tests and the 
accountability formula that is derived from the test. 

A few months after release of the KIER-sponsored Western Michigan evaluation came 
mother study of KIRIS, this one even more negative. This study was sponsored by the 
uffice of Education Accountability (OEA) a legislative agency charged with monitoring 
compliance with the provisions of KERA. In the study, a panel of six measurement experts 
had very strong criticisms of both the KDE and the private contractor employed by the state 
to develop and operate the assessment and accountability systems. Panel members believed 
that gains in KIRIS scores were "substantially inflated." Furthermore, members also believed 
that technical procedures used in developing KIRIS were seriously flawed. These included 
procedures for setting standards, for equating test forms, and for calculating test reliability. 

As with the Western Michigan evaluators, the six-member measurement panel had suggestions 
for changes in KIRIS and accountability. In some cases, the suggestions from the two 
evaluations were similar. For example, the measurement panel suggested that multiple choice 
items continue to be used in KIRIS. However, the panel suggested that portfolio scores not 
be figured into the accountability index for schools, which conflicts with the advice from the 
Western Michigan evaluators. 

A final major research study worth noting did not generate much controversy, but 
from a practical standpoint may be the most useful of all— at least at the local school level. 
The study, published in August 1995, dealt with the extent to which teachers have 
implemented instructional changes consistent with KIRIS. It was sponsored by the Kentucky 
Institute for Education Research (KIER). The statewide study involving several hundred 
classrooms revealed that teachers have altered instructional procedures to make them 
compatible with newer forms of assessment. However, teachers often do not integrate 
preparation for KIRIS with regular classroom instruction. Furthermore, there is considerable 
variability in the degree to which teachers have implemented instructional changes as a 
consequence of KIRIS. 

Where does all of this leave assessment and accountability? At the risk of using a 
cliche, the systems are in a state of flux. During December 1995, Department of Education 
personnel read proposals from potential contractors who will have bid on operating the 
assessment and accountability systems in the next accountability cycle. It is uncertain how 
many of the concerns raised by the evaluations of KIRIS will be addressed in the revised 
systems. 

Almost certainly, several issues will continue to be controversial in Kentucky. These 
include: whether to include multiple-choice items in KIRIS, what to do with portfolio scores 
(e.g., include them in accountability calculations or not), and how to estimate reliability and 
validity of the KIRIS scores and the accountability decisions made on the basis of the scores. 
There is a clear need here for continuing research by both state agencies and independent 
researchers. 
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The Primary Program 

The multiage Primary Program is probably the second most debated aspect of KERA. 

It is a prime example of the finding of several studies that "policymakers often proceed with 
reforms without understanding what teachers need to know to be able to implement them" 
(Corcoran and Goertz, 1995). Many policy makers naively believed, or at least hoped, that 
there would be a more direct and linear relationship between the promulgation of the primary 
program policy and the implementation of the reform in Kentucky’s elementary schools and 
classrooms. What research on the implementation of the primary program has shown is that 
many variables intervene between the conception and implementation of educational reform. 
These intervening variables include the knowledge structures of teachers and administrators 
who must implement the reform; t)ie existing professional culture, norms, and practice in 
schools; and student and family beliefs, perceptions, and preferences as well as the attitudes 
and beliefs of the wider public. 

Studies of the primary program have revealed a wide range of implementation with 
some teachers implementing the program with a high fidelity to recommended practices; 
whereas other teachers have failed to implement the program a" recommended. The failure of 
many teachers to successfully implement the primary program can be traced to a number of 
the intervening variables mentioned previously. First, many teachers lack the knowledge of 
the recommended teaching practices needed for successful implementation of the primary ' 
program. Although Kentucky is one of the few states that has attempted to give professional 
development a significant emphasis in its reform effort through large increases in funding, 
many primary teachers have not received the amount and kind of in-depth training necessary 
for implementation. 

Furthermore, many teachers do not find the recommended practices compatible with 
their beliefs about children and learning. These teachers often ally themselves w h parents 
and members of the broader public who cling to a view of schools that more nearly resembles 
the schools that they attended as children. They clamor for a return to "the basics" which 
usually means that the complainants believe that children should spend more time on spelling, 
grammar, and computation than on the more challenging tasks of thinking and problem- 
solving envisioned by the framers of KERA. 



Professional Development 

It will require extensive staff development before many of Kentucky’s educators will 
be able to implement the changes envisioned by the framers of KERA. Implementing the 
reforms will require a variety of development efforts including changing the way that 
educators and the public think about professional development, building support networks for 
teachers and administrators, increasing the depth of teachers’ content knowledge, and 
improving their ability to help all students meet the higher academic expectations set forth in 
KERA. Changing the way teachers teach is primarily a matter of teacher learning and even 
though structural changes such as multiage grouping or block scheduling in high schools may 
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be put in place, changes in structure will not result in improved student learning without 
providing teachers with the knowledge and support they need. 

Most educators are adamant in their belief that professional development is central to 
the success of educational reform in Kentucky. More than 70% of principals, teachers, and 
school council parents view professional development as a high priority. Legislators and 
other decision makers also recognize the importance of this effort and have increased the 
amount of both money and time allotted to professional development. One of the most 
impressive changes in the last five years is that funding for Professional Development has 
increased from $1.00 per student in 1989 to $23.00 per student in 1995-1996. Nevertheless, 
the extensive and complex nature of the changes mandated by KERA will require continued 
time and money for the requisite staff development needs of teachers and administrators. 

The nature of the professional development provided for teachers must also be 
addressed. Although perceptions about the quality of professional development opportunities 
is generally positive with over three-fourths of teachers, administrators, and school council 
parents judging staff development efforts to be working well, teachers express some concerns. 
They express the need for more time to engage in staff development activities but worry 
about reducing instructional time to engage in these activities. They also report that they do 
not have enough time to plan with colleagues and studies reveal that only about one-fourth of 
the schools had implemented some type of flexible scheduling to facilitate teacher 
collaboration in curriculum planning and professional development. 

In the section that follows, we provide summary statements about what we know and 
do not know about KERA at the end of 1995. We begin with research in the area of finance 
and follow with statements regarding governance and curriculum. We end with perceptions 
and beliefs about KERA. 
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SCHOOL FINANCE REFORM 



What We Know 



1 . In Kentucky, average per pupil revenue from state and local sources (weighted by the 
number of pupils in each district) increased by 39.4% between 1989-90 and 1993-94, 
increasing from $3,079 to $4,291. 

2. The gap between per pupil spending in the poorest districts and the wealthiest districts 
continues to narrow, indicating that the SEEK funding formula is accomplishing the 
mandated goal of equalizing educational funding for students throughout the 
Commonwealth. When comparing combined state and local resources, the difference 
between the highest and lowest wealth quintiles of districts in 1989 was $1,468 
compared to $745 in 1993-1994. 

3. On a statewide basis, 72.4% of the new funds made available under KERA were 
allocated to instruction, compared with 4.2% allocated to administration. 

4. Overall, Kentucky teachers’ salaries increased (in nominal dollars) between 1989-1990 
and 1993-1994 from $27,838 to $33,338— a 19.8% increase. The percentage increase 
was 25.4% in property-poor districts. However, average salaries in real dollars 
(correcting for the effects of inflation) increased only 6.3%. with average salaries 
continuing to lag behind those paid in other states. 

5. Surveys in 1995 showed that attitudes toward the school finance formula used in 
Kentucky are positive. Educators have quite positive feelings (67% of principals and 
51% of teachers believe the formula is "working well"). Parents also have relatively 
positive feelings (60% of public school parents and 50% of school council parents 
believe the formula is "working well"). 

What We Do Not Know 



1 . What is the cost of preparing a student in Kentucky to be a productive member of the 
information economy, and how is the cost changing over time? 

2. What evidence exists other than KIRIS scores that increased funding under KERA has 
had an impact on students’ education? 

3. What changes in the school finance reporting system are necessary so that the precise 
effects of educational reform can be measured? For example, what data are needed 
for researchers to analyze the relationships among finance data, socioeconomic data, 
and student achievement data? 

4. What are the effects of financial rewards on school performance? 
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5. What is the relationship between school expenditures in various categories (e.g., 
educational technology, professional development) and school performance? 

6. How have poorer schools used the additional funding they have received to improve 
instruction? On what have they spent money? 



SCHOOL-BASED DECISION MAKING 

What We Know 



1. Progress in implementing SBDM has been considerable with 922 of Kentucky’s 1,365 
schools having established SBDM councils. However, the rate of progress has slowed 
with approximately 100 schools establishing councils during each of the last two years. 
The remaining 450 schools must establish councils by July 1, 1996 unless they qualify 
for an exemption. 

2. Members of SBDM councils report that SBDM is working well and view the council 
as a positive change in school management, even though they find the time 
requirements of council membership demanding. 

3. As time passes, councils tend to deal with an increasingly complex array of issues. 
Early in their implementation, councils tend to focus on non-academic areas, such as 
student discipline, extracurricular activities, and facilities. Gradually they begin to 
tackle more complex issues such as budget and curricular decisions. ■ 

4. The nature and extent of participation in council decision making by parents and 
teachers varies considerably, largely as a function of the principal’s leadership style. 
Principals continue to play a major role in leadership with both principals and teachers 
assuming more leadership than parent members in many of the councils. 

5. Parent participation in running for the councils and voting in council elections tends to 
be low, especially by low income and minority parents. However, minority parent 
participation is increasing as a result of the requirement that in schools with a minority 
student population of 8% or more, parents of minority children may elect their own 
representative. 

6. Approximately 70% of principals, parent members of councils, school parents and the 
general public rate SBDM as working very or moderately well; however, only 58% of 
teachers and 40% of school board members think SBDM is working well. School 
board members express concern that SBDM has created confusion in the role of the 
school boards versus the role of the councils and that SBDM hiring practices have 
created divisiveness. 




xm 



IV 



What We Do Not Know 



1. What effects do school councils have on student learning? 

2. As councils continue to expand their decision making powers, what additional conflicts 
will arise between school councils and local boards? 

3. How can parent involvement in SBDM councils be broadened beyond the few parents 
who serve on councils and council committees? 

4. What is the impact of school councils in improvement efforts in schools in crisis? 

5. Does increased teacher and parent involvement in school management result in higher 
student achievement? 

6. Why do some schools embrace and other schools resist the establishment of councils? 
What are the most effective strategies for helping schools who have resisted councils 
develop shared decision making councils? 



REORGANIZATION OF THE KENTUCKY DEPARTMENT OF EDUCATION 

What We Know 



1. The Kentucky Department of Education was, in fact, abolished and reorganized in 
June of 1991. 

2. Five major studies (Steffy, 1992; Van Meter, 1992; Lusi, 1992, 1993, 1994) have been 
conducted on the KDE. Some of their findings include the following: 

• The KDE is still heavily bureaucratic in nature, marked by centralized control 
and top-down decision making; 

• Department personnel are working hard to implement KERA reforms. 

• Efforts are being made to increase the organizational capacity necessary for 
implementing and sustaining education reform. 

• School personnel would advise KDE to reduce regulations and paperwork, be 
clear and consistent, slow the time line, and to better understand the schools 
and districts involved in change. 

3. Systematic survey data gathered by KDE’s Office of Regional Assistance reveal that 
local districts and schools are finding the RSC’s to be valuable resources. The kinds of 
services that RSC provide vary across the state, but there is a generally high demand 
for assistance with technology, curriculum, school-based decision making, and high 
school restructuring. 
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4. Recent survey data gathered by the Kentucky Institute for Education Research (1995) 
indicate that school personnel’s satisfaction with KDE depends upon the particular 
initiative. For example, school principals, teachers, and school council parents feel 
that KDE is doing a reasonable job in improving communication between the schools 
and the Department. However, these same respondents expressed concern about 
KDE’s efforts in establishing the validity and reliability of KIRIS. 

What We Do Not Know 



1. To what extent is the reorganized KDE meeting the needs of schools as perceived by 
constituents themselves 9 

2. What have been the contributions of the Regional Service Centers to the role and 
function of KDE? What changes, if any, need to be made to increase the effectiveness 
of RSCs located throughout Kentucky? 

3. What is the quality of personnel employed by KDE relative to their defined roles? 

4. To what extent is the professional development of KDE staff consistent with KERA 
principles? 

5. To what extent is the turnover of professional and support staff within KDE caused by 
the new demands and responsibilities created by a reorganized agency? Is this kind of 
turnover to be expected and what is its effect on KDE’s ability to provide services? 



EDUCATION PROFESSIONAL STANDARDS BOARD 

What We Know 



1. Since November 1990, when the EPSB held its first meeting, the board has taken a 
number of actions including the following: 

• The EPSB has approved and implemented a professional code of ethics for 
Kentucky’s educators. 

• The EPSB has developed several guidelines for revocation proceedings 
including the Certification Revocation Statement of Purpose and Philosophy 
which outlines the grounds and procedures for revocation, and the Case 
Management Flowchart which delineates the movement of a case from receipt 
of the initial complaint through investigations and the hearing process to board 
adjudication. 
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As of June 1994, the EPSB has redefined and streamlined teaching certificates 
in seven areas including birth to primary, primary to grade 5, grades 5-9, 
grades 8-12, and in a number of specialization areas. 

The EPSB has developed and disseminated performance standards for new 
teachers, experienced teachers, and administrators. Performance standards for 
guidance counselors have also been developed. A number of subcommittees 
are currently in the process of developing level and subject specific 
performance measures in each area. Two studies dealing with the assessment 
of teachers have been completed (Cheyney University, 1994; Kifer & Guskey, 
1994). Both studies stress the need to ensure that any assessments that are 
developed and implemented are both valid, reliable, and generalizable. 

The EPSB has entered into a joint pilot project with NCATE to integrate the 
national and state accreditation standards that Kentucky teacher education 
institutions must pass to maintain accreditation. The College of Education at 
the University >f Kentucky was the first institution to be examined in this joint 
pilot project and received a favorable review in the Fall of 1995. 

The EPSB has reviewed and revised portions of the Kentucky Teacher 
Internship Program (KTIP) and brought the program into alignment with the 
New Teacher Standards. 

The EPSB has currently authorized three alternative certification programs 
which currently enroll about 50 people. To date, 32 candidates have completed 
the programs and are serving their internship year or are employed as 
classroom teachers. Research from the one program that has been formally 
evaluated (the Teacher Opportunity Program at the University of Kentucky) 
indicates general satisfaction with the program as well as two areas in need of 
improvement: 1) participants need opportunities to complete academic 
prerequisites before entering the program; and 2) there is a need for more 
African-American mentors in the program. 

The Kentucky Department of Education has several efforts to increase minority 
recruitment and retention underway including minority scholarship programs at 
Kentucky universities; minority alternative certification programs (considered to 
be joint efforts with the State Board of Education); the Advancement Via 
Individual Determination (AVID) program which is designed to help minority 
and disadvantaged students become college-bound; and TeacherB ridge, a 
summer program aimed at interesting college and community college students 
in becoming teachers. 

The EPSB has developed an initial master plan which is intended to guide the 
board’s work and provide continuity as board members’ terms expired and new 
members joined the board. 
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2. The Board’s short history has been controversial in regards to funding and governance. 
Although the members of the Board are autonomous of KDE, they still depend upon 
KDE for funding, staffing, and administration of Board policies. Board members have 
expressed concern that these differences between KDE and the EPSB can hamper the 
Board’s ability to make the changes it desires. 

3. University and college faculty, and school district personnel appear to remain 
relatively uninformed of the role and actions of the EPSB. For example, only 12 of 
the 26 colleges currently receive the board minutes and agenda though the KCATE 
e-mail list, and board staff indicate that few institutions of higher education are 
regularly represented at board meetings. 

What We Do Not Know 



1. The effects of reducing the categories of specialization for teacher certification. Will 
teachers’ preparation in specialty areas relative to their teaching assignment be affected 
significantly? 

2. The quality of teacher preparation under alternative teacher certification compared to 
traditional higher education based programs? 

3. The extent to which EPSB efforts to increase the number of minority teachers in 
Kentucky is achieving its goal? 

4. The effects of the statewide effort to develop and implement performance-based 
teacher and administrator preparation programs? 

5. The projected cost of developing and implementing a performance-based certification 
program? 

6. The validity and reliability of performance assessments developed and piloted in 
teacher education programs? 

7. The overall effectiveness of the EPSB to improve the quality of teacher and 
administrator preparation and certification in Kentucky? 



ASSESSMENT AND ACCOUNTABILITY 

What We Know 



1. A variety of assessment formats, including performance assessments, has been 

developed and administered to students in grades 4, 8, and 11/12 in all schools in 
Kentucky in the past four years. The Kentucky Instructional Results Information 
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System (KIRIS) uses multiple choice items, essay examinations, open-response items, 
performance events, and portfolios. 

2. A school accountability system that includes several cognitive components and one 
non-cognitive component of school performance was designed and implemented. 
Because of increases in their school accountability index, educators in 430 schools 
received cash rewards in 1995. Rewards totaled 26 million dollars. 

3. Based on KIRIS assessments and non-cognitive data, schools improved between 1991- 
92 and 1993-94. Accountability index scores increased 22% at grade 4, 13% at grade 
8, and 9% at grade 12. 

4. Reliability coefficients for different KIRIS measures have been calculated by the 
company contracted to develop the system. The results are that traditional assessment 
formats (multiple-choice tests, open-ended tests, essay tests) are more reliable than 
alternative formats (performance events and portfolios). In addition, KIRIS data at the 
school level are more reliable than data for individual students. 

5. Consequential validity data on KIRIS have been reported by the Kentucky Department 
of Education (KDE). Among the results are: 

• A large number of teachers received inservice education related to assessment. 

• Changes in KIRIS scores and accountability indices were relatively unaffected 
by background variables like percentage of minority children in a school, 
percentage of children on free or reduced lunch, and geographic region of the 
state. 

6. Correlations have been calculated between KIRIS and the following standardized tests: 
the Comprehensive Tests of Basic Skills (CTBS), the American College Testing 
Company’s college placement test (ACT), and mathematics tests in the National 
Assessment of Educational Progress (NAEP). The largest correlations are in the range 
of .71 to .77, which the assessment developer characterized as "modest." Many 
correlations are lower than the .71 to .77 range. 

7. Two independent evaluations of KIRIS and the Kentucky school accountability system 
have been conducted. The two studies reported a number of strengths and 
weaknesses, which included: 

Strengths 

• Individual assessment items are generally well-written and intellectually 
challenging. 

• There have been positive effects on instruction — especially increases in student 
writing, largely as consequence of writing portfolios being used in assessment. 
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Weaknesses 



• Multiple-choice items were dropped from KIRIS, but should have been retained 
in order to enhance test reliability. 

• Technical procedures used in developing KIRIS were flawed, including 
procedures for setting standards, for equating test forms, and for calculating test 
reliability. 

• Scoring of writing portfolios by teachers resulted in inflated and unreliable 
portfolio scores. 

• The decision accuracy of judgments based on the accountability index was 
poor, meaning than some schools were placed in a low category of 
performance due to chance factors. 

• Educators are skeptical of the validity and reliability of the KIRIS assessments 
and the accountability judgements based on them. 

8. A study of how teachers use performance assessment in classrooms was conducted in 
32 schools across Kentucky. It was found that teachers have altered instructional 
procedures to make them compatible with newer forms of assessment. However, 
teachers often do not integrate preparation for KIRIS with regular classroom 
instruction. Furthermore, there is considerable variability in the degree to which 
teachers have implemented instructional changes as a consequence of KIRIS. 

9. A number of statewide surveys of stakeholder perceptions and attitudes towards KIRIS 
have been conducted over the past two years. The results show that confidence in 
KIRIS as a valid and reliable measure of student performance has declined and 
confidence in other measures (e.g., standardized tests) has increased. For example, 
only 28.8% of teachers surveyed in 1995 believed KIRIS to be a very good or 
moderately good measure of how effective schools are performing (a decline in 
percentage from 1994). In contrast, 67.3% of teachers surveyed believed standardized 
tests to be a very good or moderately good measure of school performance (an 
increase in percentage from 1994). 

10. Based in part on the recommendations of evaluation studies, the Kentucky Department 
of Education issued a request for proposals to revise the assessment and accountability 
systems. Modifications in the current system are planned. For example, multiple 
choice items will be included in future versions of KIRIS. In addition, new 
assessments that will be included in future years include (a) a national reference 
assessment, which will allow Kentucky students to be compared with students 
elsewhere, and (b) continuous assessment data, from non-accountability grades, to 
assist local schools in program evaluation and curriculum development. 

What We Do Not Know 



1 . 



The extent to which improvement on KIRIS scores represents real improvement in 
students learning Kentucky’s educational goals? 



2. The extent to which students in Kentucky are learning basic skills, relative to national 
standards and norms? 

3. How well KIRIS represents the subjects being taught in schools? 

4. The best methods of estimating the reliability of the school accountability index? 

5. The best methods for improving the decision accuracy of classifying schools in the 
categories Reward, Successful, Improving, In Decline, and In Crisis? 

6. How best to help teachers, administrators and parents understand the complexities of 
KIRIS? 

7. How best to help schools use the KIRIS test data to improve instruction? 

8. The best mix of assessment formats to achieve maximum learning related to the state’s 
Learning Goals? 

9. Long-term effects of the accountability system on students and teachers? 

10. How to help teachers use performance assessment for ongoing instruction? 

1 1 . The relationship between implementation of major KERA program components and 
the improvement in KIRIS test scores? 

12. How much uncertainty should be allowed in high stakes decisions, like those about 
rewarding or sanctioning schools? 



THE KERA PRESCHOOL PROGRAMS FOR AT-RISK 4- YEAR-OLD CHILDREN 
AND 3- AND 4-YEAR-OLD CHILDREN WITH DISABILITIES 

What We Know 



1. During the 1994-1995 school year, a total’ of 15.374 preschoolers were served in or 
through the KERA Preschool Program. An additional 13,212 children were served 
with federal dollars through Head Start. 

2. The KERA Preschool Program is achieving the goal of reducing the gap between 
at-risk children and the rest of the children in their classes. Longitudinal research 
reveals that former KERA participants are scoring as well as or better than a random 
sample of their peers on a number of measures of academic progress and expectations 
for future success in school and life. 
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3. One exception to the positive results for former KERA preschool participants is that 
children in the oldest cohort who participated in the KERA Preschool Program during 
its first year of implementation (1990-1991) received lower ratings on several 
measures of academic progress and social skills than a random sample of their 
agemates. Continued study of these children will be necessary to determine whether 
this is due to a "fade-out effect" of initial positive results or to the fact that the 
preschool program was of lower quality during its first year of implementation. 

4. Children in the 1994-1995 KERA Preschool Program scored higher than a comparison 
group of income eligible peers who did not participate in the program on overall 
development in a variety of cognitive, physical', and social domains. Their teachers 
and parents also judged them to have more positive social skills and fewer behavioral 
problems after participating in the program. 

5. When primary teachers were asked t orate children who were participants in KERA 
Preschool on academic performance and their expectations for the children’s future 
success in school and in life, their ratings indicated that they expected these children 
to do as well as or better than a comparison group of children from their classes. 

6. Children with disabilities (i.e., developmental delays, speech-language problems, or 
severe disabilities) made positive gains while participating in the Preschool Program, 
indicating that the Program is addressing the unique needs of each child. 

7. The quality of KERA preschool classrooms improved substantially from 1992 to 1994 
as rated by trained observers on the Early Childhood Environmental Rating Scale. 

8. Parents of KERA preschool participants and teachers in the program reported high 
levels of satisfaction with the KERA Preschool Programs in terms of the effect of the 
program on children’s development. 

What We Do Not Know 



1 . The extent to which the initial positive gains reported for participants in the KERA 

Preschool Programs will be maintained over time? 

% 

2. The relationships among various aspects of program quality and various measures of 
student outcomes? 

3. Why more than half of the general public and school parents-at-large perceive the 
KERA Preschool Programs to be working poorly when administrators, teachers, and 
parents of preschool participants give high approval ratings to the program? 

4. The extent to which the KERA Preschool Programs are achieving the ancillary goals 
of providing integrated services to families and facilitating interdisciplinary and 
interagency collaboration among organizations serving young children in Kentucky? 
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5 . 



The effects of inclusive Preschool Programs on children with and without disabilities, 
or on the families of these children? 



PRIMARY PROGRAM 

What We Know 



1 . The nature and quality of implementation of the primary program has changed very 
little from 1994 to 1995. Significant levels of improvement have been observed in 
two components: classroom teachers’ collaboration with special area teachers and 
involvement of parents in evaluation of children’s progress. Significant declines were 
noted in two other components: flexible arrangement of the physical environment and 
provisions for children’s continuous progress through the primary program. 

2. The nature and quality of implementation of the primary program varies a great deal 
from teacher to teacher even within the same school. However, in a few schools the 
classroom observations revealed that all of the randomly selected teachers were 
implementing the program with a high fidelity to recommended practices. 

3. Overall, teachers are using recommended instructional practices in reading, writing, 
and mathematics. However, fewer than half of the teachers were using recommended 
practices in science, social studies, and the arts. 

4. Teachers who were judged to be high implementors of the primary program tended to 
achieve high ratings on components related to dynamic instruction; that is, they used 
a variety of grouping patterns to support continuous progress, they engaged students 
actively in learning, and they varied their instructional strategies to meet students’ 
needs and interests. 

5. Teachers are struggling with the implementation of continuous progress and multi-age, 
multi-ability grouping. Most teachers are employing a dual-age grouping arrangement. 
Slightly more than half of the teachers said they would retain the multi-age 
requirement if given the option to do so, but many expressed concerns regarding their 
ability to meet the wide range of instructional needs present in multi-age classrooms. 

6. Research indicates that 10% of the observed teachers had flexible, multi-age groups 
that include three or more grade levels for the entire day, 41% of the teachers had 
flexible, multi-age groups that included two grade levels for the entire day, 27% had 
less-flexible groups that included children from two grade levels for at least part of the 
day, and 15% of the teachers had other arrangements. 

7. Approximately 75% of children remain with the same teacher for two or more years. 
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8. Teachers who successfully implement the primary program tend to hold constructivist 
views of children’s learning, believing that children learn best through active 
involvement in meaningful, authentic learning activities, and find the primary program 
to be compatible with those views. Successful implementors are also committed to 
making the primary program work, see themselves as lifelong learners, and believe 
that their efforts affect children’s learning. 

9. Teachers reported having more planning time than before the primary program even 
though they expressed a need for more time to plan instruction, develop curriculum, 
and collaborate with other teachers. 

10. The aspects of the Primary Program that teachers report are the easiest to implement 
include matching learning with abilities, multi-age grouping, helping students progress 
at their own rate, performance assessment, and descriptive reporting on student 
progress. The aspects of the Primary Program that teachers report are the most 
difficult to implement are performance assessment, descriptive reporting of student 
progress, and multi-age grouping. 

11. In surveys of various stakeholders regarding their opinion of how weii the primary 
program is working, 61% of primary teachers believe that program is working well, 
whereas 41% of the intermediate, middle school, and high school teachers believe it is 
working well. In addition, 82% of superintendents, 50% of principals, 58% of school 
council parents, 62% of public school parents, 68% of the public (who claim some 
knowledge of KERA), and 41% of school board members believe the primary program 
is working well. 

What We Do Not Know 

1. The relationship between the extent of implementation of primary program 
components and student learning of Kentucky’s academic expectations, including those 
traditionally considered as "basic skills?" 

2. The effects of Kentucky’s primary program on different types of students compared to 
more traditional graded programs? 

3. Those teacher factors that contribute to, and those teacher factors that impede, the 
successful implementation of primary program components? 

4. The most effective strategies to help teachers implement developmentally appropriate 
practices, continuous progress, authentic assessment and the other critical attributes of 
the primary program? 
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HIGH SCHOOL RESTRUCTURING 

« 

What We Know 



1. During the 1994-95 school year, most high schools involved in restructuring were in a 
planning phase. 

2. Based on a study completed in 1995, schools that had the highest implementation of 
high school restructuring had the following characteristics: 

• strong leaders; 

• meaningful involvement from all major stakeholders: teachers, counselors, 
students, and parents; 

• reallocation of funds for instruction; 

• established standards for new high school requirements; 

• improved use of time by students and faculty. 

3. Innovations that are getting attention (e.g., individual graduation plan for high school 
students) will require training and technical assistance support. Based on high schools 
that were studied in 1995, several kinds of information were identified as being crucial 
to understanding restructuring, including data on: 

• enrollments, 

• expenditures, 

• governance, 

• studejnt retention, 

• student transition from school to work, 

• community involvement, 

• delivery of instruction. 

What We Do Not Know 



1. Are the changes sustained in high schools that have restructured? What changes occur 

in restructuring efforts implemented at a school? 

•« \ 

2. Do restructuring activities have an impact on student performance? For example, how 
will graduates of restructured high-schools perform after graduation, e.g., on the job, in 
college? 

3. What has been the effect of school/university collaborations on high school 
restructuring? 

4. What are the relationships between high schc 4 restructuring and the other KJERA 
initiatives? 
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5. Whal is the relationship between the extent of implementation and school 
performance? 

6. What are the impacts of the various components of restructuring on student learning? 

EXTENDED SCHOOL SERVICES 

What We Know 



1. In 1994-1995, over 150,000 students were served in ESS regular and summer school 
programs designed to help them pass a given course, achieve promotion to the next 
grade, and/or graduate from high school. 

2. Due to transportation constraints and the preferences of students, teachers, and parents, 
the majority of students are served in after school programs; although at the secondary 
level where student employment and extracurricular activities may interfere, before 
school, Saturday, and evening programs have also been utilized. 

3. Numbers of ESS students who are promoted or who graduate on time have increased 
steadily since the first year of the program as have the percentages of students who 
have improved one or more letter grades in the subject in which they were receiving 
additional instruction. Referring teachers also reported improved rates of homework 
completion, improved attitude toward school, and greater independence and 
self-regulation by the ESS students. 

4. Students in all grade levels have been served by the ESS during the first five years of 
the program. In 1994-95, for example, 54% of the ESS population was served in 
grades K-6, 26% in grades 7-9, and 20% in grades 10-12. 

5. ESS students, their parents, and their teachers all reported high levels of satisfaction 
with the program. Principals ranked ESS as the most effective KERA program with 
nearly 90% judging the program to be working well. Over 60% of the general public 
and school parents-at-large also rated the program to be working well. 

What We Do Not Know 



1 . What are the long-term effects of participation in ESS programs? Do the initial 
positive benefits affect later school achievement and successful transitions to work and 
further education? 

2. What is the specific contributions of the ESS program to school performance and 
student learning? 
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3. Do the Kentucky ESS programs meet the needs of the at-risk students in the most 
effective manner or are there other more successful models that could be employed? 

4. To what extent are Kentucky schools targeting ESS resources on the most needy 
students? 



EDUCATION TECHNOLOGY 
What We Know 



1 . The State Board for Elementary and Secondary Education has adopted a master plan 
for the Kentucky Education Technology System (KETS). In addition, a number of 
other plans and guidelines have been developed and implemented including the KETS 
Implementation Plan , the KETS Architectural Standards and Technical Specifications. 
the KETS Blueprint and Selection Guide , the KETS Building Wiring Standards , and 
the KETS Planning Workbook . 

2. Nearly all school districts have submitted district technology plans and inventories of 
each district’s "unmet needs" in technology have been prepared. 

3. Twenty-five thousand student workstations have been installed in Kentucky’s public 
schools. Local area networks and office and communication software are installed in 
central office sites in all 176 school districts, and all districts are connected to the state 
network. 

4. Significant progress has been achieved in building the capacity to make technology 
and technology networks available. For example, a recent report (Geiger, 1995) 
indicates that Kentucky is tied for seventh in the country in computers per student 
(10.2 students per computer). Kentucky also ranked seventh in the percentage of 
schools reporting sufficient computer networks, fiber optic cables, televisions, cable 
televisions, and VCRs, although a third of Kentucky’s schools indicated they did not 
have sufficient computer networks. 

5. Total expenditures in support of KETS by both state and local technology through the 
reporting period ending August 18, 1995 are $136,700,000. A draft report, Kentucky 
Education Technology System (KETS) Update, prepared by the Office of Education 
Accountability (Sanders, 1995), concludes that the unmet needs of students and 
teachers can never be resolved with the current funding level. The report states that 
$280,000,000 would have to be spent at this point if the implementation plans were to 
remain on target. 

6. Professional development options in technology remain limited. Inadequate time for 
planning and development, the lack of training and technical assistance, and the lack 
of matching funds are cited as the greatest problems. 
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7. A variety of reports indicate that approximately 20 - 25 % of teachers utilize 
instructional technology as a regular part of their instruction. Those teachers that do 
use computers often use them for lesson planning and classroom management. When 
students use computers, they often use them for learning keyboard skills, typing 
portfolio entries, remediation exercises, and drill and practice activities. 

8. A number of factors appear to facilitate the instructional use of technology. These 
factors include small class size, non-departmentalized classrooms, the extent to which 
technology and software are available to individual teachers and students, the 
availability of varied and flexibly scheduled professional development options, and a 
technology rich environment. Factors dealing with the administrative and managerial 
aspects of KETS appear necessary, but not sufficient for high implementation of 
technology in classroom instruction. These findings suggest the need for a shift in 
focus from the managerial aspects of KETS to the instructional and professional 
development aspects of KETS that are directly applicable to teaching and learning. 

9. A statewide survey conducted for the Kentucky Institute for Education Research 
(KIER, 1995) indicated that 82% of principals, 84% of teachers, 81% of school 
council parents, 61% of public school parents, and 61% of the general public feel that 
KETS is working well. A different survey by Hougland, Berger, & Evans-Andris 

( 1995) indicates that the public does support funding for increased use of technology 
in schools but that more than half of the respondents reported they were not at all 
familiar with KETS. 

What We Do Not Know 



1. What are the most effective training models that will result in the effective integration 
of technology into the management and instructional processes of schools across 
Kentucky? 

2. What support is needed for teachers and administrators that will result in the effective 
implementation of KETS? What support will be needed by teachers and 
administrators in order to maintain high levels of effective implementation of 
educational technology? 

3. What are the effects of a statewide, mandated technological intervention on student 
achievement? 

4. The relationship between extent of implementation of education technology 
components and student learning? 

5. The relative effectiveness of various software and technology applications? 
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PROFESSIONAL DEVELOPMENT 

What We Know 



1. With the exception of Fayette and Jefferson Counties, school districts across the 
Commonwealth have organized themselves into consortia to plan and implement 
professional development opportunities. 

2. Funding for professional development is based on average daily attendance and has 
increased from $1.00 in 1990-1991 to $23.00 for 1995-1996. 

3. The majority of stakeholder groups (i.e., school board members superintendents, 
principals, teachers, school council parents, public school parents, and the general 
public) that have been surveyed perceive that professional development is working 
well. 

4. Schools that are high implementors of innovative professional development options 
differ from low implementors in the effective creation and use of such things as 
mentoring, action research, local school expertise, and individual growth plans. 

5. Most schools have developed and implemented professional development plans but 
many do not have clear mission statements or stated sets of beliefs about professional 
development. 

6. Fleavy workloads of teachers and administrators constrain their opportunities to 
participate in professional development activities and to benefit from that participation. 

7. Principals, teachers, and school council parents uniformly report they believe that 
teachers and administrators need time for design of curriculum and instruction. 

8. Teachers generally report that curriculum development training opportunities are 
limited and of mixed quality. 

9. There is a need for more "content" training for both teachers and administrators. 

% 

10. Most schools continue to employ traditional professional development practices that 
occur outside the everyday routines of school and that lack the follow-up, practice, and 
reflection needed to influence actual classroom teaching. 

What We Do Not Know 

1. The quality of professional development training that administrators and teachers have 
received? 
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2. The nature of the relationship between specific professional development strategies and 
student learning? 

3. The effectiveness of Kentucky’s "market driven" statewide system of professional 
development? 

4. The impact of specific professional development strategies on various instructional and 
school management practices? 

5. The best models for providing more time for professional development within the 
school day? 

6. The effect of Kentucky’s licensing and certification system on the motivation of 
teachers to engage in quality professional development? 

7. The most effective methods for providing professional development? 

8. The nature and effectiveness of professional development provided by the Regional 
Service Centers, institutions of higher education, school consortia, and private 
vendors? 



CURRICULUM FRAMEWORK 

What We Know 



1 . The first edition of Transformations: Kentucky’s Curriculum Framework was 
developed and disseminated to local schools in June 1993. A second edition of 
Transformations has been developed and is available as of September 1995. 

2. Draft forms of the Content Guidelines for assessment in mathematics, practical living, 
reading, science, social studies, vocational studies, and writing have been distributed as 
of September 1994. Content Guidelines in arts and humanities were distributed in 
January 1995. A second version of the Content Guidelines for Assessment was 
scheduled to be released during the fall of 1 1995. 

3. High school course outlines in 15 subjects have been distributed to all public high 
schools and school districts as of July 1994. 

4. A Unit of Study Development Criteria, and A Unit of Study Reviewing Guide was to 
be distributed during the fall of 1995. 

5. The extent to which schools have changed their curricula varies enormously both 
within and among schools. Studies by AEL (1994, 1995), Corcoran (1995), and 
Matthews (1995) indicate: 
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• One of the major changes is an increased emphasis on writing and the writing 
process (AEL, 1994, Corcoran, 1995). 

• There has been a "dramatic" increase in the amount of group work, writing, 
problem solving, hands-on activities, and the kinds of higher-order questions 
asked by teachers in the mathematics and science classes, particularly in those 
classes in which math portfolios are developed for KIRIS. Traditional 
instruction - textbooks, worksheets, and lectures - still occurs with some 
frequency in most classrooms (Corcoran, 1995). 

• Preparing for the mathematics portfolios in the fourth and eighth grades takes 
an enormous amount of time and energy. In addition, one of the unintended 
instructional results of the math portfolios is that teachers spend a great deal of 
time helping students work on common prompts that later appear in the 
portfolios. While there is nothing unethical about these instructional practices, 
it does appear that the high stakes assessment system is distorting the 
instructional use of portfolios (Corcoran, 1995). 

• Teachers are using anything and everything to redesign their curriculum. 
Without textbooks and clear direction, most teachers feel frustrated and unsure 
of their efforts (AEL, 1995). 

• Students were working in pairs or groups in about half of the classrooms that 
were visited by researchers in the AEL (1994) study. Students were involved in 
hands-on activities in slightly less than half of the classrooms visited. Teachers 
in fourth and fifth grade are employing authentic literature during reading 
instruction at least as much as they employ basal readers. 

• Nine out of ten teachers in Matthews’ (1995) study reported using oral and 
written open-ended questions on a regular basis and eight out of ten teachers 
reported using portfolios tasks within units of instruction. 

• Teachers are spending more time in collaborative discussion about instruction 
and what works (Corcoran, 1995). The content and terminology of KERA, 
Transformations, KIRIS, and the Content Standards have provided teachers 
with frameworks and language for those discussions. 

6. The portfolios and open-ended responses portions of KIRIS seem to be the driving 
force behind the changes in instruction (AEL, 1994; Corcoran, 1995; Matthews, 

1995). Although most teachers have access to Transformations, KDE Content 
Guidelines, and other KDE documents dealing with curriculum, many teachers report 
that they find Transformations and other materials overwhelming, unclear, too general 
and not specific enough (AEL, 1995). 




xxx 



34 



7 . 



Teachers feel that Transformations, the Content Guidelines, and other KDE materials 
dealing with curriculum need to be more condensed, consistent, and easier to use. 



8. Teachers need additional professional development in instructional approaches that will 
help all children reach the learning goals identified in KERA (AEL, 1994). In 
addition teachers need more time to make the changes necessary to bring their 
curriculum in alignment with KERA (Corcoran, 1995). 

9. Corcoran (1995) reported that the teachers in his study felt that they needed more 
training in the subject area of science. Most of the teachers in the Corcoran study felt 
that they had adequate backgrounds in mathematics. Corcoran concludes that "Generic 
improvements in pedagogy such as cooperative learning will not make much 
difference if the teacher lacks the content knowledge to guide students to the 
understanding of underlying concepts." 

10. Schools need assistance in learning how to engage in coordinated, cohesive efforts at 
changing their curriculum in order to help student achieve KERA goals (AEL, 1994). 

11. The Kentucky institute for Education Research (KIER, 1994; Wilkerson, 1995) 
conducted statewide telephone surveys of superintendents, principals, instructional 
coordinators, school counselors, parents on school councils, parents, and members of 
the general public which indicate: 

• The majority of school professionals continue to feel positive about the 
Curriculum Frameworks. Sixty-one percent of the superintendents; 69% of the 
principals; 62% of the teachers; 69% of the school council parents; and 73% of 
the school board members felt the curriculum frameworks were working well. 

• Those professional educators who felt that the Curriculum Frameworks were 
working poorly gave the following reasons: 43% felt the frameworks were 
poorly designed; 40% felt there was a lack of time for planning and 
development; 32% felt the frameworks were not understood by teachers and 
parents; and 37% blamed a lack of training and technical assistance. 

• Helping teachers understand what students should know and be able to do at 
different grade levels; helping schools align their curriculum with academic 
expectations; and providing teachers with more time for the design of 
curriculum and instruction are identified as very high priorities by principals, 
teachers, school council parents, superintendents, and school board members. 

• Ninety-three percent of respondent groups in the 1995 survey believed that the 
performance assessments in KIRIS have changed the way teachers are teaching 
and students are learning. 
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12. A great deal of curriculum development is taking place at universities through 

initiatives like the Partnerships for Reform Initiative in Science and Mathematics 
(PRISM) project, professional associations like the Kentucky Association of Gifted 
Education (KAGE), and district consortia. In addition, the Collaborative of 
Elementary Learning is developing and distributing Different Ways of Knowing 
(DWOK) and the Kentucky Early Learning Profile (KELP). 

What We Do Not Know 



1. To what extent are schools redesigning their curricula to address KERA’s new 
Learning Goals and Academic Expectations? 

2. What specifically do teachers need in .;rder to change their curricula? How much 
guidance is needed in instructional techniques, in content knowledge? How do these 
needs vary by grade level? 

3. What kinds of information and assistance do schools, as a whole, need in order to 
develop and implement cohesive and schoolwide curricula? 

4. Do the Kentucky Curriculum Framework and other curriculum guides have an impact 
on what actually happens in the classroom? If KIRIS is, as research indicates, the 
driving force behind changes in classroom practice, what role do the Frameworks 
play? 

5. What can be done to increase the positive instructional effects of performance 
assessment and portfolios while, at the same time, lessen the negative effects on 
instruction of a high-stakes system of accountability? 

6. What is the impact of changes in curriculum on student’s learning? 

7. What are the most effective and appropriate roles of "curriculum frameworks" in the 
development of curricula? 

8. What are the most effective practices in the development of curricula? 

% 

MULTICULTURAL EDUCATION 

What We Know 



1. References to multicultural education and cultural diversity can be found in several 
provisions of KERA. The law includes requirements that teachers be sensitive to 
issues of cultural diversity, that textbooks include coverage of diverse contributors to 
society, and that efforts be made to increase the proportion of minority teachers 
throughout the state. 
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2. The Kentucky Department of Education (KDE) established the Multicultural 
Opportunities Branch in 1992. Among activities of the branch have been training 
persons to conduct workshops, developing guidelines for multicultural education, and 
supporting local efforts in multicultural education. 

3. In March 1995, a 65-member Task Force on Multicultural Education was formed. It 
consists of public school educators, professors, and researchers. The group has 
focused on three aspects of multicultural education: (a) preservice teacher education, 
(b) professional development, and (c) community awareness and support. 

4. Several minority teacher recruitment programs have been initiated. These involve 
collaborations among state universities, local school districts, KDE, and other agencies. 
Overlapping with these efforts are several alternative certification programs, which aim 
to accelerate the process by which minority persons can be employed in schools. 

5. In 1994-95, there were 44,686 certified personnel employed in Kentucky public 
schools. Of these, 1,813 ( 4 %) were classified as minority persons. The largest number 
of minority persons (1,706 of 1,813) were African-American. 

What We Do Not Know 



1. What attitudes and perceptions of multicultural education are held by students, 
teachers, administrators, school council members, KDE staff, and other key personnel? 

2. How does multicultural education affect other aspects of KERA and education policy 
decisions in Kentucky? 

3. What are the most appropriate and effective assessment strategies (quantitative and 
qualitative, traditional and performance-based) for use in studies of KERA 
multicultural components? 

4. What have been the effects of participation in multicultural inservice education on the 
classroom performance of teachers? 

^ * 

5. To what degree are the various alternative certification programs effective for 
increasing the number of minority teachers in Kentucky schools? 

6. What is the impact of efforts in Kentucky to recruit and train minority teachers? 
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KERA FAMILY RESOURCE AND YOUTH SERVICES CENTERS 

What We Know 



1. Using competitive awards procedures, 545 centers have been awarded $37 million to 
serve students and their families in 861 schools. 

2. The legislature extended the existence of the Interagency Task Force until December 
1997 to oversee the establishment and management of the centers. 

3. Several studies indicate that the centers appear to be functioning appropriately and in a 
cost efficient manner. Howe ver, in a few cases the Office of Accountability has noted 
problems in financial management and administrative practices. 

4. The centers are serving the intended population, which includes a diverse group of 
at-risk youth and their families. The needs addressed are complex and broad-based. 

No single problem predominates and participants experience several problems that 
probably interact with one another. 

5. Preliminary outcome data indicate some improvements in classroom performance 
variables as rated by teachers. 

6. In one small scale study, schools served by both a center and an SBDM council made 
greater gains on statewide assessments than did schools lacking these components. 

7. In surveys of stakeholders’ perceptions regarding how well the centers are working, 
nearly three-fourths of principals, teachers, and school council parents rate the program 
as working well. However, only half of the general public and school parents-at-large 
judge the program to be working well, suggesting that they may be less familiar with 
the positive effects of the programs on students and families. 

What We Do Not Know 



1. Will the centers improve the general well-being of children and their families? 

2. Will student achievement improve as a consequence of the support provided by the 
centers? 

3. The extent to which implementation of various components of the FRYSC program 
has occurred across the state? 

4. The impact of the FRYSC program on school performance and student learning? 

5. The impact of the FRYSC program on the "non-cognitive" factors including retention 
and drop-out rates? 
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PERCEPTIONS AND BELIEFS ABOUT KERA 



What We Know 



1. Half of the public and nearly half of the parents know little or nothing about the 
specific programs and practices introduced by Kentucky’s school reform law. 

2. Perceptions about and support for school reform initiatives are related directly to 
stakeholders’ knowledge of and involvement in schools. 

3. Public support for the learning goals defined in KERA remains at a high level. 

4. There is strong support for the beliefs associated with KERA that we should set high 
learning standards for all students and that students should be required to apply what 
they know. 

5. School professionals and parents are split on their beliefs about school accountability 
for student learning and multi-age grouping in the primary grades. 

6. A strong majority of school administrators and teachers believe schools have improved 
over the past five years: However, only about 40% of parents and the public agree 
schools have improved and about 25% think schools are worse. 

7. The most positive aspects of school reform mentioned by stake holders are increased 
emphasis on writing, increased funding to less wealthy districts, increased service 
programs to schools, and new teaching strategies; the most negative aspects include 
problems with assessment and accountability, problems related to school council, 
teacher and administrator stress, and increased paperwork. 

What We Do Not Know 



1 . What will be the pattern of public support as the reform initiatives mature? 

2. What level of public and parent knowledge about school reform is critical for the 
implementation of new programs and practices in a statewide reform effort? 

3. What are the relationships of various factors that affect polls and surveys as to their 
results? 

4. Under what conditions are the results of polls and surveys useful indicators of the 
process of reform? 
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SCHOOL FINANCE REFORM 



Stephan J. Goetz and David L. Debertin 
University of Kentucky 



What does the law or regulation mandate? 

KERA includes two key mandates with respect to school finance: ( 1 ) increasing per 
pupil funding for all students, while (2) reducing variation in per pupil revenue and spending 
across districts. KRS 157.310 specifies (General Assembly, 1990, p. 67): 

It is the intention of the General Assembly to assure substantially equal public school 
educational opportunities for those in attendance in the public schools of the Common- 
wealth, but not to limit nor to prevent any school district from providing educational 
services and facilities beyond those assured by the state supported program. 

The new school finance system is known as "SEEK"-Support Education Excellence in 
Kentucky (see also the amendment of KRS 157.320 in Section 94 of General Assembly, 
p. 67). It consists of three parts: 

1. A statewide base guarantee, or minimum foundation, of $2,495 per pupil in 1993/94; 

2. Tier I funding, which consists of locally raised funds not to exceed 15% of the base 
guarantee and which is matched with state (power) equalization funds; and 

3. Tier II funding, which allows districts to raise up to 30% of the base guarantee plus 
Tier I funding, and which is not matched by the state. 

Review of Research to date. 

To what extent has the program been implemented as intended by KERA? 

The school finance dimension of KERA has been implemented as intended: per pupil 
funding levels in nominal terms have increased in districts throughout the state (Office of 
Educational Accountability [OEA] 1994, Adams 1995a, Goetz and Debertin 1992, 1995). 
Figure 1 summarizes the dramatic effect of KERA on state and local revenues. Average per 
pupil revenue from state and local sources, weighted by the number of pupils in each district, 
increased by 39.4% between 1989/90 and 1993/94, from $3,079 to $4,291 (OEA 1994, p.46). 

These nominal funding increases do not take into account the effect of inflation on the 
purchasing power of money, however. In real (deflated) dollars, per pupil spending increased 
by 23.7% over this period (based on a cumulative rate of inflation of 12.7% between 1989 
and 1993). A single district experienced a decline in real spending per pupil between 1989- 
90 and 1993-94. In addition. Figure 1 shows that KERA eliminated the positive relationship 
that previously existed between per pupil property wealth and revenue per pupil. 
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On a state-wide basis, one important measure of funding inequality— the coefficient of 
variation (c.v.), calculated as the standard deviation of a variable divided by its mean— has 
fallen below 10% (OEA 1994; Goetz and Debertin, 1995). In general, a smaller value 
denotes less inequality. A value of 10% for this measure is considered to be an acceptable 
(or even a desirable) level of inequality in the school finance literature (e.g., Odden and Picus 
1993). However, the coefficient of variation remains at about 15% among districts in 
metropolitan areas, and at over 25% among the four districts with the largest amount of 
property wealth per pupil (OEA 1994, p. 46). 1 
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Figure 1: Total Revenue per Pupil Before and After KERA vs. 
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What have been the effects of the program on stakeholders? 

The increased funding available under KERA allowed school districts to increase 
spending in areas they considered to be most underfunded. On average, districts increased 
spending per pupil in all eight current expenditure categories (instruction, administration, 
attendance services, health services, transportation, plant operation, plant maintenance, and 
fixed costs). In relative terms, districts allocated a smaller share of the total budget to 
instruction, attendance services, transportation and plant operation and maintenance after 



'Common Core School Data analyzed in Hertert et al. (1994) reveal that 25 states had a lower pupil-weighted stai'-wide 
c.v. for per pupil revenues than Kentucky in 1989. Using Census data for unified local educational agencies. Riddle and While 
(1994) estimate that only 10 states had a lower per pupil expenditure range as measured by the ratio of the 95th-to-5th percentile 
in per pupil expenditures in 1989-90 than Kentucky, but 28 had a lower c.v. for per pupil expenditures. 
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KERA (in 1993/94) than they did before the reform, and a larger share to administration, 
health services and fixed costs (Goetz and Debertin 1995; also Adams 1995a). 

Even so, on a statewide basis, 72.4 % of the new funds made available under KERA 
were allocated to instruction, compared with only 4.2 percent allocated to administration. 
Relative to the statewide average, districts in metropolitan counties allocated a larger share of 
their funding increase to administration (4.8%), and a smaller share to transportation services 
(2.9% vs. 5.1% for the statewide average). Districts in eastern and central Kentucky allocated 
a slightly smaller share of their new funds to instruction (72.1% and 72.0%, respectively) than 
did those in western KY (73.5%) and metropolitan areas (72.8%) (Goetz and Debertin 1995). 

Costs of Teaching Courses: 

The Office of Educational Accountability (OEA) studied selected school districts to 
determine the costs of teaching individual academic courses, and providing various programs 
or services, per pupil at different grade levels, extending earlier work by Wagoner, Cotton and 
Tacogue (1993, 1994). The educational costing procedures used in the study are "designed to 
assist in budget building, increase emphasis on planning and evaluating, enhance accountabili- 
ty, and improve management" (p. 5). Using detailed 1992-93 data, the authors found that 
costs per student ranged from $2,957 to $4,309 in the eight districts studied. 

This kind of analysis is important for: 

1. identifying costs of educating students both within and across 
school districts; 

2. determining the various levels of the school system (such as 
central office administration, classroom instruction or building 
maintenance) at which actual and prorated costs are incurred; 

3. determining how these costs are changing over time with the im- 
plementation of KERA; and 

4. assessing how stakeholders at different levels of the school 
administration are being affected over time. 



Instructional Expenditures: 

% 

The OEA in its annual report categorizes districts into one of five groups (quintiles), 
obtained by ranking districts from lowest (Ql) to highest (Q5) property wealth per pupil, and 
then constructing the categories such that approximately one-fifth (20%) of all students fall 
into each category. This is useful for determining how districts with different levels of 
property wealth per pupil have responded to KERA. Perhaps the two most important 
variables in the instructional expenditure category are teacher salaries and the number of 
teachers per student, or the pupil-teacher ratio (measured as classroom size). 
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(a) Teacher salaries 



Within the instructional expen- 
diture category, total average teacher 
salaries in nominal dollars increased 
from $27,837 per teacher in 1989-90 to 
$33,338 in 1993-94 (or 19.8%) state- 
wide, and from $26,115 to $32,736 
(25.4%) in the property-poorest school 
districts (Figure 2). Higher teacher sala- 
ries may allow schools to hire more exp- 
erienced, better-trained and better-moti- 
vated teachers; so far there is no specific 
evidence that this has indeed happened. 

Considering the effects of inflation, 
however, average salaries in real dollars 
increased by only 6.3% and they 
continue to lag behind those paid in other states (e.g., Southern Region Educational Board 
1994). Average salaries paid in districts in quintile 1 are now more closely in line with those 
paid in districts in quintiles 2, 3 and 4. 

(h) Pupil-teacher ratios 

Figure 3 shows the number of 
teachers per 1,000 students increased 
from 63.2 to 64.6 between 1989-90 and 
1993-94 statewide, and from 63.8 to 
68.0 in the poorest 20% of school dis- 
tricts. This implies a reduction in aver- 
age classroom size, from 15.82 to 15.49 
pupils. In wealth quintiles 4 and 5 the 
number of teachers per 1 ,000 students 
actually declined. 

Research conducted in Kentucky 
and elsewhere suggests that test scores 
are usually, but not always, higher when 
classrooms are smaller in terms of the 
number of students per teacher (e.g., 

Berger and Toma 1994), and when teacher salaries are higher (e.g., Goetz and Luo, 1994; 
Goetz 1995a). Districts in quintile 1 (the property-poorest districts) increased both real 
teacher salaries and the number of teachers per 1,000 students by the largest percentage under 
KERA (Figure 4). On average, districts in the state in relative terms placed more emphasis 
on raising teacher salaries than they did on increasing the number of teachers per student (that 
is, on reducing classroom sizes). 
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Figure 3: Average Number of Teachers per 
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Quintile 
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Figure 2: Average Teacher Salary in 
1989/90 & 1993/94, by Wealth Quintile 
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Measuring KERA’s Impacts 



The difficulty of measuring the 
effects of KERA on educational 
outcomes to date is emphasized by 
Adams (1994). He concludes his case 
study of four school districts with the 
concern (p. 388): "... in the larger con- 
text of Kentucky’s school reform, 
conventional district expenditure data do 
not permit analysts to observe the extent 
to which reform dollars promote the 
state’s entire reform agenda." Adams’ 
recommendation is to develop a more 
refined "... financial reporting system 
that better reflects state and local reform 
efforts and better enables parents, 
taxpayers, policymakers, and others to 
measure the effectiveness of public investments in public schools" (pp. 388-9). 

Estimated Costs of Assessment Tests 

Researchers at the Center for Research in Education Finance (University of Southern 
California) have estimated the state-level costs of student assessment under KERA in a report 
prepared for the Center for Research on Evaluation, Student Standards and Testing (Picus et 
al., 1995). In addition to calculating (p. 27) state-level 1993-94 total assessment costs per 
student enrolled ($13.30), per student tested ($55.28) and per test administered ($8.86), they 
report that expenditures on assessment amount to less than 0.5 percent of total state spending 
on K-12 education (p. 25), and they conclude their study with the statement, ”[w]hat is 
surprising is, given the tremendous emphasis placed on assessment systems to measure school 
accountability, the relatively minuscule portion of educational expenditures devoted to this 
important, and highly visible, component of the educational system" (p. 29). 

The Test Score and School Funding Controversy 

One possible answer to the question of wliat effect increased SEEK spending has had 
on stakeholders is that KIRIS 2 test scores on average increased from 36.4 to 44.8 (by 23.2%) 
between 1991-92 and 1993-94, the most recent year for which data are available. However, 
considerable controversy surrounds these scores and the underlying accountability system, 
which is also discussed elsewhere in this volume. For example, one recent study concluded 
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that certain test scores from Kentucky, which could be compared nationally, remained 
constant or declined even as KIRIS scores showed dramatic improvements. 3 

A highly controversial aspect of the so-called "high stakes" accountability program 
underlying KERA, which involves increased emphasis on school outcomes as opposed to only 
inputs (such as financial resources and numbers of teachers), is the penalties and rewards pro- 
gram. Part of the controversy involves the reward funds paid based on test score results to 
better-performing schools, which may already have more resources available than do schools 
not performing as well. In addition, research suggests that factors beyond the control of 
school administrators-such as poverty rates in the school district or the level of educational 
attainment of students’ parents-influence KIRIS scores, and yet these factors are not 
explicitly taken into account when schools are rewarded or penalized (Goetz 1995a). 

Guskey (1994) warns about possible pitfalls in interpreting results of performance- 
based tests. Darling-Hammond (1994) criticizes performance-based assessment, especially 
when it is coupled with financial rewards and sanctions, because of its potentially deleterious 
effects on already poorer or lower-quality school systems. She describes a continuum of 
purposes for which assessments are used by different states. Some states (p. 20) "... envision 
carefully targeted state assessments at a few key developmental points that will provide data 
for informing policymakers about program successes and needs, areas where assistance and 
investment are needed, and assessment models for local schools." Further, (p. 20, ibid.) "... 
assessment is used as a learning tool for schools and teachers rather than as a sledgehammer 
for sorting and sanctioning." At the other end of the continuum, according to Darling- 
Hammond, lie states like Kentucky, which use performance criteria developed by outsiders 
(i.e., non-locals), and which use tests to financially reward or penalize the school system. 

Odden (1993) discusses the need to give schools more budgetary flexibility if they are 
to be responsible for school outcomes. He argues that Kentucky missed the opportunity to 
adopt new and innovative approaches to funding schools, even though (p. 311) "... the court 
certainly cleared the path to do so ...." He continues (ibid.), "[t]he opportunities that were 
missed are several, and derive primarily from the loose links between the finance structures 
enacted and the programmatic initiatives ...." Odden also writes (p. 312), "[a] finance 
structure consonant with school-based management could, at the minimum, require districts to 
provide schools with a lump sum budget, which Kentucky’s program does not do." 

Verstegen (1994) criticizes existing school finance systems, including that used in 
Kentucky (pp. 249-250): 



3 Sce Lexington Herald Leader (LHL). Tuesday. October 3. 1995, p. Bl; other recent newspaper headlines illustrate the 
controversy: "Study could fuel complaints on test scores." (LHL, Thursday, June 29, 1995, p. B3); "Tests don’t have all the 
answers to how kids rank" (LHL, Thursday, July 6, 1995, p. Al); "State might add national tests for comparison," (LHL, Wed- 
nesday, July 19, 1995, p. B4); "Concerns linger over state’s school tests," (LHL, Monday, October 9, 1995, p. Al) and "Repair 
school tests, citizens panel says" (LHL, Wednesday. October 11, 1995, p. Bl). See also the discussion in Zlatos (1995). 
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A major problem in using current school finance systems to drive systemic change and 
improvement in education is that they are obsolete and need radical restructuring if 
they are to keep pace with today’s educational needs and the global economy. ... 

School finance systems do not need to be repaired. They need to be radically 
redesigned and aligned with change and improvement across all facets of the education 
system, in an effort to achieve both excellence and equity for all children and all 
schools. 

Verstegen argues that it is insufficient to require only a minimum education for all students in 
a state, which is the main idea behind a minimum foundation program. She argues such pro- 
grams, in use during the last 70 years, were adequate for the industrial age; however, in her 
opinion, the programs do not meet the demands for educational excellence of the information 
age (p. 250). 



What are the attitudes and perceptions of stakeholders toward the program? 

Are School Administrators and Teachers Satisfied with the Results? 

A survey by Tom Wilkerson & Associates (1995) reveals that two-thirds of all prin- 
cipals believed the SEEK finance formula was "working well", and fewer than one-quarter 
believed it was "working 
poorly" (Figure 5). The 
share of principals who 
believe that SEEK is 
working well increased by 
7.8 percentage points over 
1994, while the share 
believing SEEK is 
working poorly dropped 
by 13.0 percentage points 
(this decrease was statisti- 
cally significant). On 
average, the share of re- 
spondents indicating that 
SEEK was working well 
increased across all five 
stakeholder groups, with 
the exception of public 
school parents. The oppo- 
site occurred for the 
shares of respondents 
indicating that SEEK is was 

Among teachers, 5 1 %believe SEEK is working well, up by 6.3 percentage points 
from the previous year. The number of those believing it does not work well decreased by 
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16.2 percentage points, to 22.7 percent, while the share that is undecided increased by 9.3 
percentage points (both latter changes are statistically significant). No other stakeholder 
group had a higher percentage responding that they were undecided as to whether SEEK 
works well or poorly. Of the teachers who were rewarded financially under KERA in 1995, 
58% responded that SEEK was working well. Among the teachers who were not rewarded 
financially, 47% also felt that SEEK was working well. 

Are Parents Satisfied? 

Public school parents were more likely than school council parents to respond that 
SEEK was working well (with 60.2 vs. 50.3%). These shares reflect a decline in the case of 
the former parents, and an increase in the case of the latter in terms of believing that SEEK 
works well. 

Public school parents were less likely than school council parents to respond that 
SEEK worked poorly (with 22.2% and 29.3%, respectively); among the former group more 
believed that it worked poorly than in the previous year, while in the latter group that share 
had declined (by 14.9 percentage points). 

Is the General Public Satisfied? 

Of 214 members of the general public surveyed, 58.1% believed that SEEK is working 
well (up by 9.5 percentage points from 1994), while 19.1% believed it is working poorly 
(down by 2.9 percentage points). A relatively large share of individuals surveyed were 
undecided with 22.8%, down by 6.6 percentage points. It is noteworthy that teachers as a 
group were more likely to respond that they were undecided about how well SEEK is 
working than was the case for the general public. 

Public support is essential if the state legislature is to continue to be willing to raise 
the tax dollars needed to continue to increase SEEK funding over time. The degree of public 
support suggests that continued support in the legislature may be forthcoming even as 
modification^ and adjustments to tne program are made. An important issue is the extent to 
which the test score/funding controversy outlined earlier may jeopardize future public and 
legislative support. 

% 

Another controversial aspect of the funding changes under KERA involves higher- 
spending districts. Parents in some of these districts feel they are being restricted by the 
reform act from increasing spending as much as they would like to, and this may entail an 
erosion of political support for KERA from some legislative districts (see the discussion in 
Goetz 1995£>)- The restricting effect of the reform act is also evident in Figure 1, where 
increases in per pupil spending were larger in low- than in high-property wealth districts. Of 
course, this was necessary to achieve the reduction in per pupil revenue inequality. 

At the same time, Hougland et al. report (1995, "Highlights," p. i) that "[cjonfidence 
in the ability of one’s school system to make good use of additional funds has remained at a 
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high level." Their results are based on a survey of a random sample of Kentucky residents 
carried out over a five-year period. 



What research is in progress in this area of KERA? 

Professor Jacob Adams (1995 b) of Vanderbilt University (Education and Public 
Policy) is conducting a number of studies on finance aspects of KERA. One study (1995c) 
involves the relationship between district size and state educational costs in Kentucky, and the 
question of whether school consolidation is needed. Three other studies examine the equity 
impact of school finance reform in Kentucky; the relationship between organizational context 
and district resource allocation, and how wealth, district size, poverty, and reform influence 
district spending; and the intra-district allocation of resources. 

According to Adams (1993), the general research agenda of the Consortium for Policy 
Research in Education (CPRE) includes an examination of how additional educational funds 
allocated through finance reforms are being used in various states, including Kentucky. 
Research questions of the following nature are being addressed (p. 345): ”[w]hat types of 
instructional, curricular, staff development, preschool and other programs have been created 
or expanded? What types of students-at-risk, limited-English-proficient, "average"-benefit 
from the reform dollars? How do districts and schools adapt to new governance arrange- 
ments, like site-based management? Have reforms produced fundamentally different rules for 
allocating resources to district and schools?" 

Professor Lawrence Picus (1995) of the Center for Research in Education Finance at 
the University of Southern California and his collaborators will follow-up the research on 
state-level costs of administering assessment tests in Kentucky with a study of expenditures 
and economic costs at the level of local school districts. They will attempt to determine who 
incurs the costs of testing, and what the demands on local personnel time and other resources 
will be. The researchers will also attempt to identify any benefits that can be derived from 
the administration of these tests. 

The authors of this review are continuing to investigate the determinants of KIRIS test 
scores, using socioeconomic district-level data along with data reflecting variables under the 
control }f district administrators. The method of analysis is multiple linear regression, with 
tests for district-level managerial inefficiency effects. They are also examining the 
interrelationships between student test scores and local economic development in Kentucky. 



Which questions or issues need further research? 

A number of research questions still need to be addressed: 

1 . What is the cost of preparing a student in Kentucky to be a productive member of the 
information economy, and how is that cost changing over time? Is the school finance 
system currently in place in the state adequate or appropriate for preparing students for 
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the new economy (Verstegen 1994)? This includes questions of whether the guaran- 
teed base funding amount from the state is "adequate," and whether the required 
amounts vary according to school or school-district size and other socioeconomic 
variables. 

2. What evidence other than KIRIS scores exists that the increased funding under KERA 
has had an impact on students' education? What other indicators are available to 
determine whether education is improving in general? 

3. Which schools have been most successful in raising student achievement , why have 
they been successful, and how have those schools allocated the new funds received 
under KERA ? What changes in the school finance reporting system are necessary so 
that the precise effects of reform efforts can be measured (Adams 1994)? How does 
the efficacy of reform efforts vary with the socioeconomic characteristics of the 
counties in which the districts are located? 

4. Should a state place a cap on the total amount of revenues raised by any one school 
district? If so, what should the level of the cap be? A related, more practical 
question is, are there checks and bounds in the school finance formula that will 
prevent dramatic pre-KERA revenue inequalities from re-emerging? 

5. Should financial penalties and rewards be used to improve school performance, and if 
so, how? To the extent that socioeconomic characteristics of each district affect a 
school’s ability to raise achievement levels, should those characteristics be considered 
in allocating funds? For example, if schools located in districts with low average 
family incomes have less success in raising achievement scores than do schools in 
districts with higher incomes, should an allowance be made for that fact in penalizing 
or rewarding changes in achievement scores? 
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SCHOOL-BASED DECISION MAKING 



Charles J. Russo and Jane Clark Lindle 
University of Kentucky 



c What does the law or regulation mandate? 

One of the cornerstones of the Kentucky Education Reform Act (KERA), Kentucky 
Revised Statutes (KRS) § 160.345, mandates the creation and full implementation of School 
Based-Decision Making (SBDM) in virtually every one of the more than 1,300 public schools 
in the Commonwealth by July 1, 1996. However, the law recognizes two limited exceptions. 
The first excuses school districts with only one school; the second exempts, upon request to 
the State Board of Elementary and Secondary Education, any school which is performing 
above its Kentucky Instructional Results Information System (KIRIS) Accountability 
Threshold Score as determined by the Kentucky Department of Education. 

Councils are composed of six members: two parents, three teachers (certified staff 
may not serve), and the principal or an administrator. Pursuant to changes enacted during the 
1994 legislative session of the General Assembly, in a school with a minority student 
population of 8%, measured as of the previous October 1, parents of minority children may 
elect a minority parent if one is not selected at the first election. In addition, minority 
teachers may elect their own representative. This means, for example, that the usual six 
member composition of a council may increase to eight by the addition of two minority 
representatives. Thus, the addition of minority representatives does not affect the provision 
that permits councils to increase in size as long as they maintain the same proportional 
representation. Council member: nay be elected to either one- or two year-term but are 
ineligible to succeed themselves if they are elected to a two-year term. 

Schools are given broad responsibilities under the SBDM mandate. Along with 
determining by-laws and procedures, SBDM councils select and hire principals, determine 
instructional materials such as text books, and "in consultation" with the principal assist in 
selecting staff. SBDM councils also have primary authority in eight decision areas for policy 
making found in KRS § 160.345 (2) (i) (1-9). These decision areas include the’ following: 

• Determination of curriculum 

• Assignment of time during the school day 

• Assignment of students 

• Specification of activities during the school day and week and accommodating 
the school board’s designation of the yearly calendar and beginning and ending 
times 

• Determination of instructional practices 

• Selection and implementation of discipline and classroom management 

techniques 
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• Selection of extracurricular programs and regulation of student participation in 
these programs 

• Procedures consistent with state standards and local board policy, technology 
utilization, and program appraisal (KRS § 160.345 (2) (i) (1-9)). 

Along with KRS § 160.345, five Kentucky Administrative Regulations (KARs) are in 
effect to help implement the statute. 

• 701 KAR 5:080 describes the application process necessary prior to the approval of 
alternative SBDM models. 

• 701 KAR 5:085 outlines the hearing process for complaints relating to SBDM. 

• 701 KAR 5:100 offers guidelines for alternative models of SBDM. 

• 702 KAR 3:245 provides the school council allocation formula for use pursuant to the 

Kentucky Uniform School Financial Accounting System. 

• 702 KAR 3:246 provides the school council allocation formula use in connection with 
the Kentucky Education Technology System Chart of Accounts. 

In addition to the law and regulations, the Department of Education promulgates 
Program Reviews. Although these Program Reviews are non-binding, they represent the 
perspective of the Department of Education, often in conjunction with interpretations of the 
Attorney General’s office, and should therefore be given due consideration. 

The most pertinent Program Reviews (PR) are: 

• PR No. 93-SBDM-105, Suggested Hiring Procedures in SBDM Schools , offers 
guidance to superintendents, principals, and councils as they go about the task of 
filling vacancies among their faculties and staffs. 

• PR No. 93-SBDM-l 17, PAG 93-3 1 Regarding SBDM , summarizes an Opinion letter 
from the' Attorney General of Kentucky relative to the implementation of SBDM. 

• PR No. 93-SBDM-l 19. Best Practice/Recommendations Regarding SBDM and 
Exceptional Children Services , clarifies the role of SBDM councils as they relate to 
students, programs, and faculty who are involved in special education. 

• PR No. 93-SBDM-l 20, Increasing Minority Participation on School Councils , is a 
follow-up to a change in the language of KRS § 160.345; it provides suggestions and 
strategies to increase minority participation in the SBDM process. 
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• PR No. 93-SBDM-123, School Council Policy on Assignment of Instructional and 
Non-Instructional Staff Time, clarifies the role of SBDM councils as they exercise 
their authority concerning faculty and staff assignments. 

• PR No. 93-SBDM-124, Alternative Chair Person for School-Based Decision Making, 
under which SBDM councils are given the latitude to elect a chair of their choice. 
Previously, the principal alone had the authority to chair a SBDM council. 

• PR No. 93-SBDM-125, School-Based Decision Making Council and Committee 
Meetings Open to the Public, explains the application of KRS § 61.810, the Open 
Meetings Law, to meetings of councils and their committees as duly constituted public 
agencies. 

• PR No. 93-SBDM-129, Best Practices/Recommendations Regarding SBDM an d 
Preschool Programs Including Collaborative Head Start Programs, provides 
clarification about the role of SBDM councils with regard to both KERA-funded and 
federal preschool programs. 

• PR No. 94-SBDM-136: Legislative Changes in Minority Representation . . . Structure 
and Membership Policy for Committees . . . Eligible Voters in Parent Elections and 
Terms of Office . . . , addresses the provisions added to the statute by the 1994 
session of the General Assembly. 

• PR No. 95-SBDM-149, Requirements for Implementation of SBDM and Methods for 
Exemption from SBDM Mandate in KRS § 160.345 and 703 KAR 4.080. provides a 
time line for schools that have yet to implement SBDM councils, a copy of the form 
for councils seeking an exemption from the law, and information about potential 
sanctions for schools that do not comply. 

• PR No. 95-SBDM-150, Relationship Between School-Based Decision Making Councils 
and Family Resource/Youth Service Centers (FRYSCs). clarifies the relationship 
between SBDM councils and FRYSCs. 

Finally, the Attorney General of Kentucky has prepared approximately 30 non-binding 
opinion letters (OAG) to clarify the operation of the law. The complete list of OAGs can be 
found immediately following the statute in the 1994 edition of Kentucky School Laws 
Annotated. 

Under the law and accompanying regulations, SBDM councils have extensive policy- 
making authority relating to the governance of local schools. Among the powers delineated 
in KRS § 160.345, some are shared in conjunction with the local board of education. 
However, in Board of Education of Boone County v. Bushee ( Bushee ) (1994), the Supreme 
Court of Kentucky affirmed that there are many areas of the statute where the authority of the 
councils are not subject to approval or rejection by the board. 
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Review of Research to date. 



To what extent has the program been implemented as intended by KERA? 

As of October 9, 1995, 922 of Kentucky’s 1,365 schools have formed SBDM councils. 
Six sites are exempt from the SBDM mandate since they are the only schools in their 
districts. Eleven schools that had councils in place have been granted exemptions for 
exceeding their KIRIS Accountability Threshold. In addition, approximately 70 schools that 
have yet to implement SBDM have requested exemptions; their petitions will be presented to 
the State Board of Education in June of 1996. 

Presuming all these schools are granted exemptions, the remaining 367 schools are 
encouraged through Program Review 95-SBDM-149 (April, 1995) to implement the following 
steps to meet the legislature’s July 1, 1996 deadline: 

Obtain a successful %’s vote of the faculty to implement SBDM 
Hold elections for all council members 

Receive training and establish schedules and record keeping procedures 
Establish committee structure 

Develop and adopt by-laws and School Transformation Plan 
Begin work on SBDM policies as outlined in KRS § 160.345 (2)(i) 

Prepare budget for professional development, instructional materials and 
personnel policies (PR 95-SBDM-149, p.2) 

After July 1, 1996, all these responsibilities accrue to the SBDM councils of the late- 
adopting schools with the exception of the %’s vote of the faculty (PR 95-SBDM-149, p.8). 

Studies of SBDM implementation have primarily been descriptive as researchers have 
attempted to document the process of implementing SBDM councils in the early adopting 
schools. There has been little research on non-SBDM schools. 

Various researchers have studied specific implementation issues at SBDM schools in 
the following areas: (1) leadership in voluntary and early adoption of SBDM, (2) training 
influences on implementation, (3) the extent to which councils have dealt with the various 
decision areas allocated to school councils under KRS § 160.345 (2)(i), and (4) the extent to 
which curriculum and instruction have been addressed by councils. One additional study 
looked at the influences on school district policy in developing SBDM. 

There are two long-term studies of district and school implementation of SBDM. Jane 
David of the Bay Area Research Group is in the last year of a five-year study of SBDM 
implementation commissioned by the Prichard Committee (David, 1992, 1993, 1994a, 1994b). 
The other long-term study is the Appalachian Education Lab’s (AEL) in-depth case studies of 
four rural Kentucky school districts’ implementation of KERA (AEL, 1991; Coe & Kannapel, 
1993; Coe, Kannapel, Aagaard & Moore, 1995; Kannapel, Moore, Coe, & Aagaard, 1994). 
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Implementation of SBDM began with school districts’ responses to KERA directives 
for establishing SBDM implementation procedures (KRS § 160.345 (2)(a); KRS § 160.345 
(3)(a)-(i)). Babbage’s (1993) dissertation described the influences of the Kentucky School 
Boards Association (KSBA) on school districts’ implementation of this mandate. He analyzed 
all Kentucky districts’ SBDM policies for their congruence with KSBA’s model policy. Then 
Babbage chose four districts as prototypically similar or dissimilar to the KSBA model. He 
conducted in-depth interviews with policy makers in these districts. He concluded that 
districts felt pressed for time in policy development in response to KERA which may have 
limited the policy-making process. 

Questions about barriers to implementation of SBDM have been an ancillary focus of 
SBDM studies which include non-SBDM schools. The Kentucky Department of Education’s 
Division for SBDM used two annual surveys of SBDM implementation to include both 
SBDM and non-SBDM schools (Kentucky Department of Education, 1993; Lindle, Gale, & 
Curry-White, 1994). During the summer of 1994, the Kentucky Institute for Education 
Research (KIER) and the Kentucky Association of School Superintendents (KASS) surveyed 
superintendents on a variety of KERA issues (KIER & KASS, 1994). Adams-Rogers (1995) 
also completed a case method history of KDE’s role in SBDM implementation. 

Although the majority of superintendents in the KIER and KASS survey reported that 
SBDM was working well, many complained about the poor implementation, management and 
design of SBDM (KIER & KASS, 1994, p.7). Adams-Rogers found that time constraints were 
commonly cited as a barrier to SBDM, but satisfaction with the status quo also was an 
impediment to SBDM implementation (Adams-Rogers, 1995). In the KDE surveys, non- 
SBDM principals were polled for reasons that schools had not implemented SBDM and ways 
that KDE could provide support services. In 1993, the top three reasons non-SBDM principals 
gave for not yet adopting SBDM were time constraints, lack of information, and lack of 
training (KDE, 1993, p.10). In Fall 1994, KDE asked both non-SBDM principals and 
superintendents to give reasons that schools had been reluctant to adopt SBDM. Both non- 
SBDM principals and superintendents indicated that the demands of implementing other 
KERA mandates, time constraints, and faculty resistance were barriers to SBDM 
implementation. Significant differences were noted between the superintendents’ and non- 
SBDM principals’ responses in 1994. Non-SBDM principals were more likely than 
superintendents to ascribe time constraints as a barrier. Superintendents were more likely to 
assign strength to the principals’ influence on the’decision to implement SBDM. Non-SBDM 
principals’ and superintendents’ responses split significantly over the superintendents’ and 
school board members’ influence on the school’s decision to implement SBDM with 
principals ascribing more influence to both groups than superintendents (Lindle, Gale, & 
Curry-White, 1994, pp. 20-24). These school leaders’ contrasting responses indicate a need 
for studies of leadership in schools which have and have not adopted SBDM. 

The leadership of the principal has emerged as a critical factor in voluntary adoption 
and implementation of SBDM (Coe, Kannapel, Aagaard, & Moore, 1995; David, 1994a; 
Prichard Committee, 1995). Several dissertations from the University of Kentucky have 
explored various principal leadership characteristics in relationship to SBDM Council 
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members’ satisfaction with their activities (Gerl, 1994; Hutton, 1995; Schadler, 1994). 
Principals of elementary, middle and high schools reported more coaching or supportive 
leadership styles than directing or delegating styles and were shown to have consensus with 
parent and teacher council members’ perceptions of principal leadership styles. For high 
school principals, a measure of leader effectiveness was found to be a predictor of SBDM 
council member satisfaction (Gerl, 1994). This was not true for elementary principals, but the 
school’s achievement of KERA’s learner outcomes was found to be a predictor (Hutton, 

1995). Among middle schools, Schadler found that women were significantly less satisfied 
with SBDM council achievements and activities than males (1994). 

Another implementation issue has been training. In order to assist with implementation 
of SBDM, the Kentucky Department of Education (KDE) has established a network of 
approximately 100 endorsed trainers. The trainers are required to attend an annual two-day 
workshop sponsored by the KDE in order to maintain their status as state-endorsed. An early 
study of SBDM training (Donelan, 1992) reported that while 80% of the SBDM Council 
members reported training in the law relating to SBDM, most reported little training in the 
actual processes of policy development and decision making granted under the law. Based on 
the 1994 Kentucky Department of Education survey of SBDM school council members, 
superintendents, and non-SBDM principals, more than 70% of the respondents had made use 
of the SBDM Trainer’s network. The majority of these respondents rated the network as 
"Good" to "Excellent." Even so, the 1994 study concluded that KDE should move from 
training focused on legalistic compliance to training that is more supportive of policy and 
decision-making processes (Lindle, Gale, & Curry-White, 1994). The effectiveness and 
appropriateness of SBDM training was questioned in a study by the Kentucky Institute for 
Education Research (KIER, 1995). This study of the nature and extent of SBDM 
implementation focused on 3 1 schools randomly selected from the 8 service regions of the 
Commonwealth. KIER found that "the appropriateness of SBDM training and training 
frequency were ... subcomponents least correlated with overall SBDM implementation" (p.4). 
The best predictor of SBDM implementation was the degree to which a school was 
implementing and evaluating any type of school plan (School Transformation, School 
Technology, Chapter 1, or Extended School). 

KIER’s study is one of several that address the importance of the decision areas 
allocated to school councils under KERA (KRS § 160.345 (2)(i)). Nearly all implementation 
studies comment on the tendency of SBDM councils to focus on non-academic and 
procedural matters in the early phases of implementation (David, 1992, 1993, 1994a, personal 
communication August, 1995; KIER, 1995; Martin, 1995; Prichard Committee, 1995). The 
1994 KDE study of SBDM reported virtually no difference between SBOM and non-SBDM 
schools in addressing the decision areas as they related to developing School Transformation 
Plans; however, School Council members’ awareness of their KIRIS Accountability Threshold 
scores were related to the implementation of decision areas. Members who reported higher 
scores also reported directly addressing more of the decision areas in SBDM Council (Lindle, 
Gale & Curry-White, 1994). A dissertation by Read (1994) of 51 SBDM schools also 
revealed limited progress in addressing all of the decision areas. He further reported that 
often SBDM councils delegated these responsibilities either formally or informally to 
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principals (Read, 1994). Appalachian Education Laboratory (AEL)’s studies show a different 
assumption of these decision areas by SBDM councils. The AEL studies report that SBDM 
councils which are dominated by educators or principals make fewer decisions in the KERA- 
designated decision areas (Coe and Kannapel, 1993; Kannapel, Moore, Coe & Aagaard, 1994; 
Coe, Kannapel, Aagaard, & Moore, 1995). AEL researchers point out that the progress of 
SBDM implementation is not a narrow linear path (Coe, Kannapel, Aagaard, & Moore, 1995). 
A study by Martin (1995) of eight secondary schools in central Kentucky supports AEL’s 
observation on the non-linearity of council development. Martin reviewed first-year minutes 
of the eight schools, and then observed two schools whose council minutes reported more 
decisions on curriculum and instruction. She found that neither SBDM Council fit the typical 
stages of group development; instead the complexity of agenda items influenced their 
processes (Martin, 1995). 

A few implementation studies on the decision areas are more specifically related to 
curriculum and instructional practices. Studies conducted by Logan (Logan, 1992; Logan & 
Byers, 1995) focused on instructional and curriculum decisions for vocational education under 
SBDM. These studies showed that SBDM had a negligible effect on the integration of 
academic and vocation curriculum, but that vocational and academic secondary teachers did 
report a closer working relationship from 1992 to 1994 (Logan & Byers, 1995). 

Studying the implementation of SBDM on a statewide scale is a daunting task. 
Nonetheless, implementation studies have revealed these rather consistent findings: 

• Time is perceived as an important resource for policy making and 
implementation of SBDM by school districts, and both early and late-adopting 
schools. 

• Principals are key leaders in encouraging and supporting the implementation of 
SBDM. Some evidence suggests that principals may underestimate their own 
influence on SBDM processes. 

• Training has been another important implementation component, but the focus 
on the legal requirements of SBDM may not have given SBDM councils 
enough decision and policy making process information. 

• SBDM councils’ attention to policy development may relate to several factors. 
SBDM councils which focus on the School Transformation Planning processes 
of implementation and evaluation and/or School councils engaged in addressing 
KIRIS results and their Accountability Thresholds may be more likely to 
develop policies related to curriculum and instructional practices. Not 
surprisingly, SBDM group processes may be an influence on the council’s 
capacity to address substantive instructional or curricular issues. 



What have been the effects of the program on stakeholders? 

Presumably, students would be the primary stakeholder group impacted by SBDM. 
Defining measures of SBDM effectiveness for students is problematic even at the national 
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level (Malen, Ogawa, & Kranz, 1990). One reason that student effects are so removed from 
school-level decision processes is that typically these structures are initiated to provide a new 
level of professionalism for teachers rather than to provide improved student learning 
(Carnegie Forum, 1986; Ogawa, 1994). So far, no studies of Kentucky’s SBDM have directly 
addressed student effects, and a few hint that it may be too early to measure student effects 
(Association for Supervision and Curriculum Development, 1994; Coe, Kannapel, Aagaard, & 
Moore, 1995; David, 1994a; KIER, 1995). 

There also have been limited studies of SBDM effects on teacher professionalism. An 
early study by Rogers (1992) found that elementary SBDM teacher representatives were 
inexperienced in decision-making processes and viewed the selection of a principal and 
instructional materials as the most important areas of involvement for SBDM councils. 
Another study by Smith (1995) found that professionals in schools with councils reported the 
closest correlation between their ideals of how a school should operate and the way schools 
do perform. On the other hand, Rinehart, Short, and Johnson (1993) studied Kentucky and 
Alabama teachers’ feelings of empowerment and found no difference between teachers in 
SBDM schools over non-SBDM schools. However, there were significant differences between 
Kentucky and Alabama teachers (Rinehart, Short & Johnson, 1994). 

Effects of SBDM on stakeholders have not been well documented. While defining 
effects is an empirical problem for SBDM researchers, clearly there is room for further 
research in this area. 



What are the attitudes and perceptions of stakeholders toward the program? 

Many studies are concerned with comparing and contrasting various stakeholder 
perceptions. Perceptions and attitudes of stakeholders have been solicited through surveys 
and interviews. In general, results reveal that SBDM appeals to all the involved stakeholders 
with the possible exception of those school boards which have filed direct challenges to the 
scope of the SBDM portion of KERA (David, personal communication, August, 1995). The 
majority of stakeholders in the schools are of the opinion that the SBDM is working well 
(Wilkerson & Associates, 1995) and are familiar - with the provisions of the SBDM mandate 
(Appalachia Educational Laboratory, September 1991). 

t 

Power issues emerge from inquiry on perceptions. Lindle (1992, May, September) 
compared pilot year (1991-92) council members’ satisfaction with SBDM council 
communication. She found principals to be significantly more satisfied with communications 
than either teachers or parents. Parents were the least satisfied (Lindle, 1992, September). 
Logan and Byers (1995) also reported that secondary principals were more satisfied with 
SBDM than teachers or counselors. Sigafus (1994) depicted one secondary faculty’s struggles 
with democratic discourse in policy development, and Lindle (1994) described the 
micropolitical tug-of-war between the faculty and the principal. Unsolicited comments from 
respondents to the KDE’s 1994 SBDM survey predominantly focused on power issues and 
conflicts with principals over agenda-setting and information (Lindle, Gale & Curry-White, 





1994) . Principals are seen as more powerful than teachers because the principals evaluate 
teachers, and teachers hold power over parents through expert knowledge and evaluation of 
the parents’ children (David, 1994a; personal communication, August, 1995). The AEL 
studies identified three distinct power structures in councils: (1) advisory to the principal, (2) 
teacher- or principal-dominated, and (3) balanced where all participate in discussions and 
decisions (Coe and Kannapel, 1993; Kannapel, Moore, Coe & Aagaard, 1994; Coe, Kannapel, 
Aagaard, & Moore, 1995). Martin’s (1995) research suggests that the demographic 
homogeneity of councils may actually work to prevent SBDM councils from engaging in 
constructive conflict or dealing with wider ranges of alternatives for the decision process. 
Gender differences may also affect satisfaction with council activities and communication and 
thus merit further study (Schadler, 1994; Lindle, 1992, September). 

A perennial issue identified in various studies is parent involvement. The Prichard 
Committee studies (David, 1994a) and the AEL studies (Coe, Kannapel, Aagaard & Moore, 

1995) as well as a poll conducted by the Louisville Courier-Journal (Schaver, 1994) illustrate 
a general concern about low parent participation. Lindle (1992, May) found that SBDM 
councils used a very limited variety of methods to reach and communic£ l parents 
although there were more effective ways to communicate. The 1994 KD-, survey also 
showed that SBDM Council members were quite frustrated about getting parents involved 
(Lindle, Gale, & Curry-White, 1994). The ICIER study (1995) supported other studies’ 
findings that parents are minimally involved in SBDM activities. Specific case studies, 
however, show a large parent turnout when the SBDM council addresses substantive issues 
(Lindle, 1994; Sigafus, 1994). 

Although there is general agreement that SBDM is a workable governance reform, 
studies of stakeholders’ perceptions reveal areas of concern. Two of these areas, allocation of 
power and parent involvement, may have some demographic antecedents. The following 
demographic variables may have important effects on stakeholder perceptions of council 
efficacy and satisfaction with council deliberations and activities and need to be studied 
fuither: 

• role or position of the stakeholder, 

• gender, race, ethnicity, and/or socio-economic status of the stakeholders, and 

• diversity among SBDM members. 



What other questions have been addressed? 



The remainder of the literature on SBDM is descriptive. Early pieces focused on the 
history and key provisions of the law (Van Meter, 1991; Russo, Donelan, & Van Meter, 
1992; Steffy, 1993; Herman & Herman, 1993; Noland, 1991; Olasov, 1991). Other legal 
analyses have resulted as challenges to the law have been raised or settled (Russo, 1995a, b, 
1994-95; Russo & Harris, 1995). 





Treatises on how to interpret and implement SBDM provisions were also common in 
the early literature. The Kentucky School Boards Association commissioned a handbook by 
Harvey (1991), and the Prichard Committee provided a handbook by Weston (1993). Weston 
also contributed a handbook on school budgeting which SBDM councils have found useful 
(Weston, 1992). The most recent handbook is a 500 page document, Synergy , by the 
Kentucky Department of Education (1994b). Another useful handbook gave suggestions to 
SBDM councils on how to address the needs of exceptional children through PR-93-SBDM- 
1 19 (Interdisciplinary Human Development Institute, University of Kentucky, 1995). 

Other early articles offered guidelines for implementation (Grover, 1992; Nystrand, 
1991). Magruder (1991) described issues for training SBDM councils. Lindle and Shrock 
(1993) reported on one secondary school’s SBDM Council’s policy on hiring staff under KRS 
§ 160.345 (2) (h). 

The more recent work has documented SBDM progress in Kentucky to a national 
readership (Association for Supervision and Curriculum Development, September 1994; 

David, 1994b; Lindle, in press; Russo, Van Meter, & Johnson, in press; Van Meter, 1994). 



What research is in progress in this area of KERA? 

1. David, Jane L. (in progress). Jane David continues her five year study on the 
implementation of KERA for the Prichard Committee for Academic Excellence. 
Results should be forthcoming in the Spring of 1996. 

2. The Appalachian Education Laboratory study of four rural districts continues. 
Typically these reports are issued in the spring and summer. Thus, the next reports are 
due in 1996. 

3. Kentucky Department of Education, Division of School-Based Decision Making, (in 
progress). The SBDM survey for 1995 should be forthcoming in the winter of 1995. 

4. Office of Educational Accountability (in progress). The 1995 Annual Report should 
be forthcoming in the fall of 1995. 

% 

5. Several doctoral students in the Department of Administration and Supervision at the 
University of Kentucky are exploring various issues in school governance under 
KERA. Six of these works in progress will be concluded over the next year to 
eighteen months. 



What are the questions, issues, or needs for further research? 

The research studies conducted to date can hardly capture the diversity or definitive 
patterns represented by a statewide reform of school-level governance. Indeed, these studies 
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have provided provocative glimpses of issues in implementation and raise even more 
questions about the purposes, emphases, and impact of changing school governance. Some of 
these questions might include the following: 

• What are the similarities or differences between early and late adopting schools in 
terms of student effects, teacher professionalism, and/or power issues? 

• As schools have adopted SBDM at different rates are there any clearly identifiable 
patterns of SBDM implementation, development, and/or levels of maturity? (This 
easily might be accomplished as a follow-up to the KIER (1995) study.) 

• With the principalship identif: ;d as a key role in SBDM implementation, can patterns 
of exemplary principal leadership be identified and/or taught? Can exemplary 
principals be used as role models or mentors for other principals new to SBDM? 

• Case studies of SBDM councils employing high levels of involvement in the decision- 
making process should be conducted to identify how various constituencies are 
involved. Specific strategies for engaging parents and under-represented groups should 
be defined and explored. 
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REORGANIZATION OF 

THE KENTUCKY DEPARTMENT OF EDUCATION 



Eddy Van Meter 
University of Kentucky 



What does the law or regulation mandate? 

State law, KRS 156.016, directs that "[effective at the close of business on June 30, 
1991, all employment positions in the Department of Education shall be abolished and the 
employment of all employees in the positions will be terminated." The law further stipulates 
that, "After a comprehensive study of the Department of Education and the goals and duties 
of the commissioner of education and the department... the commissioner shall reorganize the 
department effective July 1, 1991." Finally, the law states that "[t]he reorganization of the 
department, shall incorporate a strong orientation toward providing technical assistance to 
school districts." 



Review of Research to date. 

To what extent has the program been implemented as intended by KERA? 

The most obvious initial questions relating to the regulation to reorganize the 
Kentucky Department of Education have to do with matters of fact. These questions involve 
such issues as: Did, in fact, the reorganization take place and was it accomplished in the 
timeframe specified in the law? Similar questions can be raised about whether a 
comprehensive study of the department was, in fact, conducted prior to determining the final 
reorganization design, and whether or not the new department is now organized in a manner 
that emphasizes technical assistance to the 176 school districts in Kentucky, and the some 
1,366 schools located within these districts. 

Other questions can be raised with reference to the implementation of the regulation, 
but these questions move to a level of inference about the ultimate intent of the law, and 
consequently such questions are much harder to specify in any kind of definitive manner. 
These questions relate to such issues as whether the number of department employees 
terminated was appropriate and adequate, whether adequate design alternatives for the 
department were considered, and whether the final design is a meaningful departure from the 
previously existing department structure. 

A final question of reasonable importance is to decide when the KDE reorganization 
effort has been completed since it is now more than four years after the initiation of the 
effort. At some point it would seem to make sense to no longer focus KDE-related research 
within the reorganization conceptual framework, but rather to anchor the inquiry in some 
other manner. In effect, a case can be made that in the near future so many structural and 
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personnel changes will have been made within KDE that it will be inaccurate to think of the 
agency as reflecting the circumstances of the original 1991 reorganization. 



What have been the effects of the program on stakeholders? 

At least five public manuscripts have been prepared which speak, in some manner to 
the Kentucky Department of Education reorganization effort, and address aspects of the 
factual and inferential questions noted above. These documents include the following: 

1. Steffy, Betty E. (1992). Assault on the bureaucracy: Restructuring the Kentucky 

Department of Education. International Journal of Education Reform , J_> (1). 
16-31. 

2. Van Meter, Eddy J. (1992). Restructuring a state education agency: The Kentucky 

experience . Charleston, WV: Appalachia Educational Laboratory, Occasional 
Paper Series. 

3. Lusi, Susan Follett. (1992). Systemic school reform: The challenges faced by state 

departments of education . Paper presented at the 1992 Annual Research 
Conference of the Association for Public Policy Analysis and Management. 

4. Lusi, Susan Follett. (1993). Systemic school reform: The changes implied for SDEs 

and how one department has responded . Paper presented at the 1993 Annual 
Meeting of the American Educational Research Association, Atlanta, Georgia. 

5. Lusi, Susan Follett. (1994). The role of state departments of education in promoting 

and supporting complex school reform . Doctoral dissertation, Harvard 
University, May, 1 994. 

A brief summary of each of the above documents is provided below, and particular 
emphasis is given to examining the extent to which each of the documents addresses the 
implementation questions already identified. 

There have also been several additional studies conducted in the past two or three 
years that, while focusing on different KERA issues, do include information regarding the 
operations of the restructured Kentucky Department of Education. A brief commentary about 
the content of these studies is provided after the more extensive summary of the five p r imary 
documents. 

The article written by Betty E. Steffy (1992) is a descriptive and interpretive narrative, 
covering the events that took place within the Kentucky Department of Education during the 
six month period immediately prior to the June 30, 1991 abolishment of the old department, 
and during the period immediately after the newly reorganized department was established. A 
detailed overview is provided about the manner in which a study of the department was 





conducted in anticipation of establishing a new structure for the department. Attention is also 
given to how changes in department personnel were accomplished, including the employment 
of new top-level administrators within the reconfigured department, many of whom were 
hired from outside the state of Kentucky. The issue of repositioning the department to be 
more responsive to technical assistance needs of school districts is not really addressed, and 
the general conclusion of the author regarding the reorganization is rather pessimistic 
("...where a bureaucracy has existed before, it was resurrected again with somewhat different 
functions, more aligned with the legislative language, but a bureaucracy nonetheless."). The 
article is informed by the author’s personal involvement as a top-level administrator in the 
department prior to the reorganization, which provided access to internal documents, records 
of meetings, and personal communication with department personnel. 

The paper by Van Meter (1992) also utilizes a descriptive and interpretive format of 
presentation, relying primarily on available internal source documents and public reports of 
the events leading up to the abolishment of the old department and the establishment of the 
reorganized department. Van Meter provides considerable background information relating to 
the overall design of the Kentucky Education Reform Act of 1990, and traces the procedure 
used in hiring the new Kentucky commissioner of education, in addition to describing the 
structural changes made in the department reorganization. He also in the paper does provide 
some comment regarding the development of a technical assistance thrust in the new 
department, as noted in the excerpt below: 

One particularly interesting part of the discussion in the department 
reorganization was how exactly to create a system of decentralized service 
centers, an aspect of the restructuring that was discussed early in the effort. 

There was speculation, for example, that something akin to the intermediate 
unit concept of regional education service center or area education agencies 
would be implemented as part of a much more decentralized Kentucky 
Department of Education. However, as seen in the new organization chart, a 
decision was made to establish a number of regional centers, but these are 
clearly subsumed within the centralized system and are relegated to a divisional 
status that is at least two steps removed from the office of the commissioner. 

(p. 14) 

Van Meter in concluding his paper also poses several questions that other states might 
consider if state education agency restructuring is being discussed: 

1 . Is restructuring in a nonbureaucratic manner really viable for a state education agency, 
given the responsibilities involved? 

2. Is it a good practice to employ new people in all higher-level administrative positions 
within a reorganized or restructured agency? 

3. What does it really mean for a state agency to shift from a regulatory to service 
emphasis? 
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4. When state agency restructuring does take place, how is success of the effort to be 

determined? 

Using the Kentucky restructuring effort as an example, possible answers to each of the 
above four questions are explored. This paper, like the article by Steffy, documents that a 
study of the existing department was made prior to developing a new design for the Kentucky 
Department of Education. It also reinforces Steffy ’s acknowledgment of the reorganization 
being done within the timeframe specified by statute. It does not provide the kind of detail 
that Steffy’s article does regarding the internal discussions taking place as the about-to-be- 
abolished department attempted to respond to the requirements of KERA, but it does speak 
more directly to the issue of whether or not the reorganized department is designed to 
emphasize technical assistance. In this latter regard, it should be noted that establishing 
regional centers for the agency is obviously not the only way to foster a technical assistance 
mission, and in fact a case can be made that with the establishment of a total new Division of 
Learning Support Services— which was done in the reorganization— the ability to provide 
technical support services to school districts was greatly enhanced. Thus the point made in 
the Van Meter paper was not so much that a new emphasis on technical assistance did not 
receive attention, but rather that one particular way of accomplishing such a focus was 
considered but was ultimately not implemented on a large scale. 

In the first cited paper by Lusi (1992) (Systemic school reform: The challenges faced 
by state departments of education) , her basic premise is that state departments of education 
(SDEs) must develop new internal and external working relationships to bring about the type 
of systemic school reform now being promoted by some reformers. The external 
relationships she feels are most important are those related to the ways in which SDEs need 
to work with schools in supporting decentralized, school-based efforts to improve students 
learning. She contrasts these new relationships with the traditional "top-down" relationships 
that SDEs maintained with schools in the past. Furthermore, Lusi contends that SDEs must 
also develop new internal relationships that support the changing roles of SDE employees 
engaged in support of systemic reform. The hierarchical, bureaucratic structures of traditional 
SDEs were appropriate for the predictable, routine, and repetitive nature of the regulatory 
work SDE employees often performed in the past; however, the new roles of SDE employees 
require much greater internal collaboration around a set of tasks that are less proscribed and 
more responsive to the needs of constituents. Lusi points out that such work requires a 
different kind of SDE administrative structure, one that is more coherent and cooperative and 
less bureaucratic. 

Although Lusi, in this first paper, cites a few examples from what she calls her 
"preliminary work" in Kentucky and Vermont, her paper speaks more to how state education 
agencies should rethink structure and functions than it does to identifying specific ways in 
which the Kentucky effort actually accomplished such changes. In effect, this paper does 
little to help the reader discover if the reorganized Kentucky Department of Education has 
been successful at establishing the types of relationships she feels are needed for sustaining 
systemic education reform. This is essentially a theoretical policy paper by design, and not 
an empirical research piece. 
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The second cited paper by Lusi (1993) (Systemic school reform: The changes implied 
for SDE’s and how one department has responded) is a much different effort than her first 
paper. If the first paper can be characterized as developing a theoretical statement about how 
SDEs would need to change in order to be responsive to a new agenda of systemic education 
reform, the second paper is an in-depth examination of the reorganized Kentucky Department 
of Education with the focus on trying to see if the new department meets the guidelines set 
forth by Lusi. The methodology employed by Lusi in writing this second paper was to 
conduct extensive interviews with numerous Kentucky Department of Education staff 
members, as well as interviewing individuals outside the department. 

At the outset of the paper Lusi restates the set of changes that one would expect in a 
reorganized SDE that was designed to support systemic reform. The five broad categories of 
change include: 

1. The substantive work of SDEs would change. 

2. The necessary organizational capacity would need to be in place. 

3. Organizational flexibility and responsiveness would increase and be prized. 

4. The organizational focus would be on building capacity in the field to meet ambitious 
learning outcomes. 

5. There would be strong, vibrant, collegial, collaborative connections with other 
institutions such as the legislature, districts, universities, and other education groups. 

Having set forth the basic features of a reform-oriented SDE, Lusi then uses Kentucky 
as a specific example of an SDE that has reorganized with an intent to be the kind of new 
SDE described. In the paper Lusi examines the specific circumstances of the Kentucky 
Department of Education with reference to each of the five categories of change already 
noted. The final conclusion arrived at by Lusi is perhaps best conveyed by reproducing a part 
of her overall summary: 

What is happening in the KDE is consistent with the framework in some areas. 

The substantive work of the department has changed considerably and efforts 
have been and are being made to put the necessary organizational capacity in 
place. ...More interesting, however, are the places where the framework and 
what is happening in the KDE diverge. The differences between the two are 
strongest in part three of the framework that examines organizational flexibility 
and responsiveness: To date, the organization appears hierarchical in both 
formal and informal structure. ...(p. 32) 
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She goes on to make a final summation: 

Implementing systemic reform, in this case KERA, is about more than simply 
implementing the law, which quickly translates into meeting the legislated 
timelines. It is also about implementing the law in such a way that deep and 
lasting reform takes place throughout the system.. ..Unless this broader task 
definition is understood and accepted, there will probably be little investment in 
conversations working toward new paradigms of how the state should interact 
with schools. Without reaching these new paradigms the likelihood of 
achieving true systemic reform seems slim. (p. 48) 

In this second paper by Lusi the questions being raised have more to do with what 
was earlier called issues of fact about meeting the specific mandates of the regulation. It is 
almost a given in Lusi’s paper that a reorganization of the Kentucky Department of Education 
has taken place, and it can perhaps be implied that it took place in the timeframe required. 
What Lusi is doing here is moving to the next set of questions, those which begin to ask if 
the department after the reorganization is designed to support systemic reform, and if the 
department is actually going about the business of such support. Recognizing that it is still 
very early in the history of the KDE transition, it is nevertheless fair to say that Lusi’s paper 
questions whether a completely adequate SDE change has taken place in the Kentucky case. 

Finally, we turn to the third and most recent contribution by Lusi (1994) (The Role of 
State Departments of Education in Promoting and Supporting Complex School Reform) . This 
manuscript, the culminating report of a completed doctoral dissertation submitted at Harvard 
University in mid-1994, continues the inquiry' of Lusi’s previously cited documents. Her 
emphasis is a detailed examination of how state departments of education (SDEs) can 
reorganize in the effort to support what has come to be called complex or systemic statewide 
school reform. Lusi selected Kentucky and Vermont as example SDEs to study since each 
was involved in an extensive systemic reform effort, and each was attempting to restructure to 
meet the challenge. 

With reference to the Kentucky case, this manuscript provides a more comprehensive 
treatment of the subject than does her previous papers. It includes more documentation of 
interviews with key KDE personnel; it includes an extensive commentary about problems 
facing state education agencies engaged in complex reform; and it includes lessons for future 
consideration based upon both the case study experiences in Kentucky and Vermont. 

Lusi sets forth the overall agenda for her dissertation by defining four research 
questions: 

1 . Given the existing literature on systemic reform, bureaucracies and innovative 
organizations, what changes would wc expect to see in SDEs successfully 
implementing complex change? 
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2. How do the expected changes in SDEs compare to what is actually occurring in the 
cases of Kentucky and Vermont, and how can the differences between what is 
occurring and the expected changes be explained? 

3. How do the SDE’s actions seem to be affecting local implementation of complex 
reform? 

4. What broader lessons can be drawn from the analysis of these two cases for SDEs 
engaged in complex school reform? 

Lusi discusses five major changes that she believes should be evident within a state 
education agency whenever the agency has taken on the task of supporting complex (i.e., 
KERA-like) reform. She maintains that included among these changes should be an increased 
agency flexibility for SDE personnel to collaborate across traditional program or regulatory 
areas of responsibility, less focus within the SDE on “following rules and procedures” and 
checking internally to ensure that initiated actions are approved higher in the bureaucracy, and 
more focus statewide on building capacity for reform among school district personnel. 

The third chapter of Lusi's study, 79 pages of manuscript, presents her case study of 
the restructuring of the Kentucky Department of Education and her examination of changes 
made in an effort to support Kentucky's KERA version of systemic reform. In the case study, 
which was completed in 1993-94, she contends that the work of the KDE did, in fact, change 
and that the new priority “...has become implementing KERA” (p. 42). With reference to a 
new KDE internal and external organizational capacity being in place, Lusi concludes that 
more attention was being given to providing state school personnel with training, professional 
development, and technical assistance. However, she found that the new KDE internally was 
not itself characterized as a “learning community engaged in internal capacity enhancement, 
but rather was faced with a pressure and stress producing circumstance such that, as described 
by one KDE director, ‘...there was never any time to stop and actually think about what you 
are doing”’ (p. 48). Lusi goes on to say that the “...formal structure of the KDE remains 
hierarchical” (p. 48), and she also found “mixed messages” being sent to school district 
personnel by KDE staff (p. 55). Furthermore, she reports that: 

After interviewing numerous staff members of the KDE, the picture that 
emerges is that of an organization primarily driven by centralized control and 
top-down decision making, as opposed to that of an organization driven by 
vision and mission with decision making authority broadly dispersed, (p. 56) 

Lusi then turns to her examination and assessment regarding KDE's support of reform 
implementation. She based her findings on an admittedly small sample of visits and 
interviews in three selected school districts. With reference to KDE oversight, Lusi noted the 
following: 

Nearly half of the practitioners interviewed (21 of 47) said that the KDE is 
overregulating the implementation of KERA, and that too much time is spent 
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on reporting on process and filling out paperwork. While virtually all 
practitioners interviewed expressed strong support for the goals of the reform, 
they described an implementation process hampered by excessive rules and 
oversight. Some practitioners felt that the department was even more 
regulatory than it had been in the past. (p. 79) 

However, she also added a caveat to the above observation by noting that even the most 
vociferous critics believed that department staff were working hard to implement the KERA 
reforms (p. 83). 

Lusi found KDE to be actively engaged in trying to build local capacity in schools 
through professional development initiatives (p.87), although she noted that much of this at 
the time of the study was within KERA strands rather than across strands. When Lusi asked 
school personnel what advice they would offer to KDE regarding support for the 
implementation of KERA, she received four general answers: (1) reduce regulations and 
paperwork; (2) be clear; (3) slow the timeline; and (4) know the schools and districts you're 
working with (p. 91). 

In summarizing the case, Lusi suggests that the KDE appeared to be more centrally 
controlled than ever, with internal staff creativity and initiative thus being stifled (p. 99). She 
also concluded that the restructured KDE was placed in a difficult situation at the outset since 
the KERA reform was already well started prior to the reorganization, and thus the agency 
was placed in an immediate catch-up posture (p. 101). Finally, she observed that the KDE 
was faced with competing demands and expectations including those of school personnel who 
wanted more freedom of operation to implement the reform, versus those of the state 
legislature which wanted more accountability for implementation and financial expenditure. 
Such competing demands, she contends, need to be clarified at the beginning of a complex 
state reform effort if state education agencies in the future are expected to play an effective 
role in the support and implementation of reform. 

What are the attitudes and perceptions of stakeholders toward the program? 

While the five documents just reviewed highlight those studies most directly relating 
to the KERA-initiated reorganization of the Kentucky Department of Education, in the past 
two or three years there have also been several additional studies which have touched on 
selected aspects of the new KDE. These have included relatively small-scale internal studies 
conducted by KDE, usually to assess client reactions to services provided, as well as larger 
research efforts initiated by other organizations attempting in some way to assess the impact 
of KERA. 

An example of the first kind of study being referred to above is a Quality and 
Importance Survey conducted for the 1994-95 school year by the Regional Service Centers 
(RSC) Office of KDE (Kentucky Department of Education, 1995). This was an internal KDE 
effort to survey school administrators, teachers and other clients served by the eight Regional 
Service Centers established as part of the new KDE design. Results of this survey were quite 
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favorable, with RSC services rated as being "very helpful" regarding such issues as providing 
information relating to KERA implementation, providing assistance with planning, and 
providing resource materials. Respondents also indicated that service demand within the RSC 
areas of the state varied, but there was a generally high demand for RSC services with 
reference to the topics of technology, curriculum, and school-based decision making. In 
addition, an increasing demand for RSC services was reported by respondents relatin 6 to the 
topics of transformation planning and high school restructuring. Over 90% of the respondents 
receiving services also indicated that the sen/ices were timely, accurate, and professional. 
Obviously there are limitations to any internally initiated evaluation-of-services study, but this 
effort and other similar studies conducted by KDE staff do provide some feedback from 
school clients regarding how well the activities of the reorganized offices of the state 
education agency are being received in the field. 

The second kind of study providing an indirect indication of how the newly 
reorganized Kentucky Department of Education is being perceived by school personnel 
throughout the state are research efforts conducted by various organizations in an effort to 
assess the impact of KERA as it is being implemented. An example of this kind of study is a 
Statewide Education Reform Survey conducted in 1995 and prepared for the Kentucky 
Institute for Education Research which has a mandate to evaluate the ongoing status of KERA 
(Wilkerson & Associates, 1995). This study included a survey of school principals, teachers, 
and school council parents from throughout Kentucky and asked, among other things, for 
respondents to rate twelve KERA-related programs and practices that potentially need 
attention in the future. While several of the twelve issues implied some possible action by 
the KDE (e.g., "Reducing the paperwork associated with KERA"), only one issue specifically 
mentioned KDE by name (i.e., "Two-way communication between schools and the 
Department of Education"). This latter issue, or practice, was rated eleventh in importance by 
principals, tenth in importance by teachers, and eighth in importance by school council 
parents. In effect, these kind of findings at least imply that the newly reorganized KDE is 
doing a reasonable job of interacting with school clients. However, since respondents to this 
particular survey did express concerns for several KERA issues that do fall within the 
purview of KDE responsibility (e.g., "Establishing the validity and reliability of KIRIS") it 
would be misleading to suggest that school personnel are completely satisfied with all KDE 
efforts and activities relating to KERA implementation. 



What research is in progress in this area of KERA? 

As of late 1995, there are no known additional research studies being conducted with 
specific reference to this topic. However, it should be stated that numerous internal KDE 
documents do exist and are continuously being created which speak either directly or 
indirectly to issues regarding the reorganization of the department. The RSC Quality and 
Importance Survey briefly reviewed in a previous sections is a good example of one kind of 
internal KDE document that informs this topic. In addition, there are various kinds of 
research efforts being conducted each year that focus on specific programmatic strands of 
KERA and on evaluations of the overall KERA initiative, and it is common for issues relating 
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to KDE to be addressed in some manner as a part of these studies. The KIER-sponsored 
Statewide Education Reform Survey also briefly reviewed in a previous section is an example 
of this kind of effort. In these efforts the intent is not to examine the circumstances of the 
reorganized KDE as such, bu* often the results do provide information that has an indirect 
bearing on the topic by reporting how school practitioners and others are responding to the 
new KDE. 

It is also important to remember that the Kentucky Department of Education is 
structurally dynamic rather than static organization. Changes in staff, and in functional 
office responsibilities, are continually taking place. The reality is that the department is not 
in some respects organized in the same manner it was when the five primary studies cited in 
this report were completed, nor is it likely that the department will remain as it is at present. 

An obvious case in point regarding the changing circumstances of KDE involves the 
resignation of Dr. Thomas Boysen as the first appointed Commissioner of Education in 
Kentucky as of the end of June. 1995, and the subsequent employment of Dr. Wilmer S. 
Cody to be the new Commissioner of Education. On August 10, 1995, the Kentucky Board 
of Education announced the appointment of Dr. Cody, with an understanding that he would 
start work in Kentucky on September 11, 1995. This transition has now been accomplished, 
Dr. Cody has officially taken on the responsibilities of the Office of the Commissioner which 
includes administration and oversight of the Kentucky Department of Education, and his 
tenure in helping to shape the future of KERA is beginning. What actions may take place 
with reference to making adjustments or changes within the Kentucky Department of 
Education under Dr. Cody’s leadership are obviously yet to be determined. 



Which questions or issues need further research? 

There are a number of issues regarding the KDE reorganization that need further 

research and examination. Further research is needed to: 

1. Obtain continuing assessments of how well the reorganized Kentucky Department of 
Education is meeting the needs of school districts and school persons, as viewed by 
the constituents themselves, with reference to KERA implementation. 

2. Assess the extent to which the most recent changes made in KDE are contributing to 
moving the department more toward a service and systemic reform support system. 

3. Assess exactly what kinds of activities and services are being engaged in by each of 
the newly created divisions of KDE, and whether professional and support staff within 
each of the divisions perceive that they are receiving adequate professional 
development to operate their respective offices in a manner that supports the overall 
agenda of KERA. 




40 




4. Determine the extent to which present and continuing turnover of professional and 
support staff within KDE is caused by the new demands and responsibilities created 
by a reorganized agency and the resultant changes 6f job activity, scope, stress and 
expectations that are a result of such a change. 

5. Assess the extent to which adequate progress is being made with reference to hiring 
minority personnel within the reorganized KDE, which has been established as a 
priority goal by the agency. 

6. Assess the impact created by appointing a new Commissioner of Education in 
Kentucky with reference to how the Kentucky Department of Education is managed 
under new leadership, and what changes of organization and mission may be initiated 
within KDE by the new commissioner over the next two- or three-year period of time. 
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EDUCATION PROFESSIONAL STANDARDS BOARD 



Elizabeth Nelli 
University of Kentucky 

What does the law or regulation mandate? 

KRS 161.028 of the Kentucky Education Reform Act established the Education 
Professional Standards Board (EPSB) and authorized the board to regulate the certification of 
school personnel in Kentucky. The General Assembly charged the board with creating a 
professional code of ethics, with reducing and streamlining the credential system to allow 
greater flexibility in staffing schools while maintaining standards for the competence of 
school personnel and for district-initiated alternative preparation programs, and with designing 
a district-initiated alternative preparation program to prepare primary teachers. Within the 
context of regulating certification, the board sets standards for the preparation of school 
personnel, standards for the approval of each preparation program, and standards for the 
accreditation of each preparing unit (school, college, or department of education) in colleges 
and universities. 

The board has sole power to issue, renew, suspend, and revoke certificates of school 
personnel. Additionally, it has been charged with maintaining data and with reporting on the 
quality of professional preparation programs. The board is also required to study the problem 
of the declining number of minority teachers and to recommend ways to recruit more 
minority teachers, develop guidelines to follow upon receipt of an allegation of sexual 
misconduct by an employee certified by the EPSB, and receive training on the dynamics of 
sexual misconduct of professionals. 

The governor appoints 13 of the 15 board members: eight classroom teachers, two 
administrators (one of them a school principal), two college or university representatives of 
professional preparation programs, and a local school board member. There are also two ex- 
officio voting members: the commissioner of education and the executive director of the 
Council on Higher Education. 

The General Assembly chose to locate staff support and funding for the EPSB within 
the Kentucky Department of Education (KDE). Thus the board’s staff, which is employed by 
the KDE to work in the Office of Teacher Education and Certification (OTEC), is evaluated 
through the KDE and reports to the commissioner of education, who has the authority to 
assign KDE staff as he or she deems best. 
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Review of Research to date. 



Overview 

Since November 1990, when the EPSB held its first meeting, the board has taken 
multiple actions to address its various charges. Its accomplishments include the following: 

• a professional code of ethics 

• streamlining the number and types of certificates in Kentucky 

• regulations to permit district-initiated alternative certification programs for high 
school, middle school, and primary teachers 

• convening of committees to work on performance standards for new and 
experienced school personnel 

• a pilot project with the National Council for Accreditation of Teacher 
Education (NCATE) to design a new protocol for the accreditation of 
professional education units 

• a workshop for board members and staff on revocation of certificates, including 
information on sexual misconduct of professionals 

• employment of hearing officers and case workers to expedite revocation 
investigations and hearings 

• procedures and a protocol for revocation hearings 

• revocation, suspension, or dismissal of more than 200 certification cases 
(compared to two cases during all of 1988, before the board was created) 

• establishment and staffing of the Division of Minority Educator Recruitment 
and Retention within OTEC 

• creation of a plan and structure for implementation during 1995 of the federal 
program to employ and prepare retired service personnel as teachers ("Troops 
to Teachers") through the U.S. Defense Activity for Non-Traditional Education 
Support (DANTES) 

• encumbrance of funds for assigning staff to work with the Educational Testing 
Service (ETS) and the Interstate New Teacher Assessment and Support 
Consortium (INTASC) on national pilot programs to ensure that the preparation 
of school personnel in Kentucky is 

congruent with national standards and trends 

• a memorandum of agreement with the commissioner of education outlining the 
working arrangements between the two and the responsibilities of each to the 
other and to KERA 

• a master plan to guide board actions and priorities in coming years 

However, very few research reports have been published in relation to board actions. 
Existing reports and monographs relating to the EPSB reflect the national trend relating to 
autonomous professional education bodies— that is, they are summaries of board actions and/or 
authority, or are opinions about specific board actions. 
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Thus the following summary is predominantly a review of board actions relating to its 
mandated responsibilities; available research literature is cited where pertinent. 



To what extent has the program been implemented as intended by KERA? 

It appears that the EPSB has moved steadily, if cautiously, to implement the charges 
of the General Assembly. The board’s Master Plan (1995) summarizes the status in each 
mandated area and outlines board priorities. The Master Plan is a major source for the 
following section. 

Professional Code of Ethics 

In May 1991, the EPSB approved the code of ethics, the first of its kind in the 
Commonwealth, and incorporated it into regulation. Revisions of the code were approved 
January 1995. The code delineates the ethical responsibilities of Kentucky’s educators to 
students, parents, and the education profession, and is the foundation upon which the EPSB 
bases its deliberations in matters of certification revocation. 

Revocation 

Beginning in 1988, new statutes and regulations penalized district superintendents who 
failed to report dismissal with cause of certified personnel. The EPSB has determined that 
intensive training in revocation procedures and parameters will be required for all board 
members, but turnover in EPSB membership has lessened constancy in the adjudication of 
cases. To increase consistency, the EPSB during the past year approved a Certification 
Revocation Statement of Purpose and Philosophy, which outlines the grounds and procedures 
for revocation, distinguishes between EPSB administrative hearings and criminal court trials, 
and defines the nexus of EPSB authority. Also approved was a Case Management Flowchart, 
which delineates the movement of a case from receipt of the initial complaint through 
investigations and the hearing process to board adjudication. 

Streamlining Certification 

In 1990, Kentucky had 156 different certificates for school personnel. In June 1994, 
by regulation (704 KAR 20:670), the EPSB redefined teaching certificates in the following 
seven areas: 



birth to primary 

primary through grade 5 ("elementary") 

grades 5-9 ("middle school"), with at least two specializations from 
English/communications, mathematics, social studies, science 
grades 8-12 ("secondary school"), with at least one specialization from English, 
mathematics, social studies, biological sciences, physical sciences. Other 
specializations may be added to the required major. 
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• grades 5-12 specializations in agriculture, business and marketing education, 
home economics, and industrial technology 

• primary through grade 12 specializations in art, communication disorders, 
foreign language, health, hearing impaired, learning and behavior disorders 
(LBD), moderate and severe disabilities (MSD), music, physical education, 
school media librarian, and visually impaired 

• additions (endorsements) to existing certificates in computer science, English as 
a second language, gifted education, driver education, and reading/writing 

A new certification handbook is currently being written, with a streamlined format 
intended to make it less technical, more readable, and therefore more useful to stakeholders. 

The new certificates become effective in 1997; how they will affect teacher 
preparation remains to be studied. Preparing institutions are currently wrestling with revising 
their programs, which must not only reflect the streamlined certificates, but must also be 
performance-based. 

Additionally, the current Program of Studies, whicn dc eribes the approved curriculum 
in Kentucky public schools and delineates the certification required to teach each grade level 
or course, does not reflect the streamlined certification levels. The KDE has delayed 
revisions to the current Program of Studies, which was written in 1986. unfil the state’s high 
school restructuring pilot project is completed in 1996. As a result, school districts are 
making many teacher hiring and placement decisions on a case-by-case basis, tying up OTEC 
staff with ongoing requests for individual placement approvals. 

Teacher Standards and Assessment 

As noted in the November i993 and December 1994 Review of Research on KERA, 
the EPSB in 1992 received a U.S. Department of Education grant of $750,000 over three 
years for the Teacher Assessment Project. The Teacher Assessment Project is intended to 
bring about the complete reform of educator preparation and certification in order to align it 
with the goals of KERA by the year 2000 (the initial completion date was 1996, but the 
complex nature of the project resulted in a revised timeline, as described in the December 
1994 Review of Research). The new system will consist of performance -based licensure for 
all school personnel as well as a revised program approval and unit accreditation process 
focusing in part on performance standards. The project also encompasses aligning state 
performance standards with national standards such as those of INTASC (1992), professional 
organizations, and NCATE. 

In fall 1992, the EPSB appointed four councils to develop performance standards for 
new teachers, experienced teachers, administrators, and guidance counselors. The board 
approved and disseminated performance standards for new teacners in 1993 (revised and re- 
disseminated in 1994), and for experienced teachers and administrators in 1994. 

Concurrently, 15 subcommittees have been developing level- and/or subject-specific 
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assessment measurements in each area, and are continuing to field-test the results of their 
efforts. 



In November 1993 Teacher Assessment Project funds enabled the KDE to award 
Cheyney University of Pennsylvania a contract to conduct an evaluation of the second phase 
of the project, specifically evaluation of assessment items, field tests, stakeholder input, 
accomplishment of project expectations, and cost analysis. Results of that evaluation, the 
Preliminary Summative Report, Kentucky Teacher Assessment Project (1994) were reported 
in the December 1994 Review of Research on KERA. Recommendations from the report are 
currently being implemented, and many are consistent with a separate in-state assessment of 
the new teacher performance standards (Kifer & Guskey, 1994). 

An unpublished quantitative analysis of the new teacher assessment tasks (Kifer & 
Guskey, 1994) identified "no apparent systematic difference between results of the portfolio 
versus the on-demand tasks" (page 3). The report discussed four different areas within the 
data set: social studies, special education, vocational education, and science. Given the 
limitations imposed on the study by the different ways data were collected, the authors 
concluded that agreement among raters is "reasonable," but that generalizability across raters, 
tasks, and respondents is limited (p. 11). They also recommended slightly different designs in 
order to obtain more definitive information (pp. 10-11). 

It is worthwhile to note that with the turnover in EPSB membership, leadership, and 
OTEC staff, communication among all the councils and subcommittees, between the 
councils/subcommittees and the EPSB, and between the EPSB, its staff, and constituent 
groups, has been sporadic at best, resulting in periods of confusion and consternation. Only 
in the past six months, in large part because of the EPSB’s Master Plan (see below) and 
continuity in OTEC leadership, have board members, OTEC staff, and interested education 
constituents apparently begun to share a common sense of purpose and direction. 

Accreditation of College/University Education Preparation Programs 

Early in its existence, the EPSB determined that quality control in the preparation of 
teachers, administrators, and other school personnel involves two processes: assessment of 
the candidates and assessment of the preparation program. As stipulated in committee reports 
the EPSB accepted and approved in 1994 (including a report from the Continuous Assessment 
in Teacher Education Working Group, and the University Policy Decisions Report) both 
processes must be performance-based and must reflect the standards established for 
professional educators. As recommended to the board, and as described in the December 
1994 Review of Research on KERA, subject-specific and level-specific continuous assessment 
processes are being developed and the board expects that these will be folded into the 
accreditation process. 

The EPSB has entered into a joint pilot project with NCATE to integrate national and 
state accreditation standards, starting with the University of Kentucky accreditation visit in 
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November 1995 and continuing through 1998 with other Kentucky institutions seeking to 
maintain national accreditation. 



The state/NCATE pilot study was conceptualized by members of the Kentucky 
Association of Colleges for Teacher Education (KACTE) and presented to the board, which 
approved the concept and general outline as well as choice of the University of Kentucky as 
the initial institution for the pilot. The former executive secretary of the EPSB negotiated 
with NCATE, and then left his position due to illness before the details were shared with 
board members or with the institutions involved in the pilot. Thus there is continuing 
confusion among institutions and current OTEC staff about expectations regarding the 
alignment between state performance standards and national accreditation standards, and the 
amount of discretion the board and NCATE will grant to institutions involved in the pilot. 

The pilot proposal submitted by KACTE included continuing assessment of procedures and 
impact, with subsequent pilot institutions to be informed of assessment results and provided 
with OTEC staff assistance in preparing for their portion of the project. OTEC staff seemed 
uncertain about preparations for implementing this part of the proposal. 

Where and how alternative certification programs fit into the accreditation pattern has 
not been discussed with preparing institutions. 

Kentucky Internship Programs (KTIP and KPIP) 

KTIP, the Kentucky Teacher Internship Program, is a legislatively mandated state 
support system for the first-year teacher, and may also be viewed as a school-based validation 
of the assessment process for new teacher candidates. During 1993-94, committees of the 
EPSB revised the KTIP observation instrument, the scoring rubric, and the portfolio 
requirements, aligning them with the newly developed New Teacher Standards for Preparation 
and Certification, as described fully in the December 1994 Review of Research on KERA and 
summarized for the EPSB in March 1995 through the KTIP Revision/KTIAC Joint Meeting 
Minutes. Revising of procedures, instruments and training of classroom observers for KTIP 
continues. 

The board formally announced its intention to revise the current internship regulation, 
and OTEC staff have been directed to that assignment. The Kentucky Teacher Internship 
Advisory Committee (KTIAC) recommended a review of current training tapes prior to 
setting new scores for observer trainees, with the purpose of evaluating the tapes’ congruence 
with the New Teacher Standards for Preparation and Certification. 

KPIP, the Kentucky Principal Internship Program, also legislatively mandated, 
continues to operate without substantive change, although two committees are currently 
meeting to update the test bank of questions (which implies a revision of the knowledge base) 
and review the observation instrument, again with the purpose of examining congruence with 
the new Administrator Standards for Preparation and Certification. 
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Alternative Certification 



In 1990 the General Assembly established basic eligibility requirements for persons 
pursuing teacher certification via a route other than the typical college preparatory program. 
Alternative certification programs must be initiated by local school districts which must seek 
cooperation with college/university faculty. The EPSB has currently authorized the operation 
of three such programs: a Kentucky State University and University of Louisville partnership 
with Jefferson County Public Schools; the Teacher Opportunity Program (TOP), a 
collaborative program involving the University of Kentucky and Fayette County Public 
Schools; and a cooperative program between Northern Kentucky University and its adjacent 
school districts. 

These programs currently enroll about 50 persons. To date, 32 candidates have 
completed the programs and are serving their internship year or have successfully completed 
the KTIP experience and are employed as classroom teachers. Alternative certification has 
been a priority of metropolitan areas, where critical masses of potential teachers could be 
identified. The existing function of all three programs currently is to recruit and prepare 
teachers of color. 

A 1994 evaluation of TOP was described in detail in the December 1994 Review of 
Research on KERA. A 1995 evaluation (Bliss, Brennan & Mack) reported continuation of 
candidates’ perception that they were in a rigorous and valuable program. Responding to the 
1994 evaluation, the program has allotted additional time for candidates to complete academic 
prerequisites and other course work to ensure that they are fully prepared in the subject areas 
taught in the elementary grades, and have had the time and experiences needed to become 
reflective practitioners (which is the UK model). The 1995 study reports positive results. 

Thus the program now is not as accelerated as originally conceived by state legislators and 
many KDE staff; however, the number of applicants to TOP has increased despite the 
additional time allotted for TOP candidates to complete the program. The researchers made 
no attempt to generalize from the UK experience to other alternative programs. 

If evaluations of the other alternative certification programs have been done, they are 
not readily available and do not appear in the research literature. This lack of evaluative 
research encourages the EPSB to rely on those who think any person holding a baccalaureate 
degree, regardless of the major and academic grades obtained, needs only a little professional 
preparation in order to be fully prepared to be a teacher in a Kentucky classroom. The UK 
study suggests otherwise, but there are no studies of the other programs to permit comparison, 
nor to validate or invalidate the commonly held assumption that life experiences compensate 
for prior academic deficiencies. 

The DANTES program was scheduled to start operating in 1995 with about 20 retired 
military persons. OTEC staff were unclear about whether the program is, in fact, in operation 
and if so, the names of currently involved districts. The EPSB has delegated a staff person 
the responsibility of coordinating and overseeing the implementation of Troops to Teachers in 
Kentucky. 





In 1991 the board committed itself to periodic on-site evaluations to verify the quality 
of alternative certification programs (704 KAR 20:600), but procedures and policies for these 
visits have not yet been implemented. 

Minority Recruitment and Retention 

The OTEC’s Division of Minority Educator Recruitment and Retention coordinates 
three efforts: scholarships with the state universities, minority alternative certification 
programs (considered to be joint efforts with the State Board of Education), and 
TeacherBridge with high schools and community colleges. A fourth initiative, AVID 
(Advancement Via Individual Determination), was listed by the previous commissioner as a 
strand within this division, but actually is coordinated through a different KDE office. 

The EPSB’s goal is that by July 1, 1996, the number of minority teachers in public 
schools will have increased by 20 per cent. ' c 

The Minority Scholarship Program, alternative certification programs, and 
TeacherBridge were detailed in the December 1994 Review of Research on KERA. The 
Division of Minority Educator Recruitment and Retention’s Report of Activities (1995), 
Annual Report (1995), and Minority Educator Recruitment Guide (1995) update and expand 
this information. In addition, the Report of Activities (1995) provides employment data by 
school district for the prior school year: available positions, number of minorities who 
applied for each position, number of minorities interviewed for each position, number of 
minorities hired for each position, and total positions filled. While these data appear to be 
fertile ground for research, no studies have as yet been reported to the board, nor do any 
appear in the research literature. 

Master Plan 

As EPSB members’ terms expired and they were replaced by other appointees, those 
members remaining recognized the need to have a master plan to guide the board’s work and 
to provide continuity. It was intended that the plan should be a living document to be 
reviev/ed and revised periodically, and that it would provide historical perspective on previous 
board actions as well as direction for future actions. The initial master plan was developed 
primarily through a staff member of the Council on Higher Education over a 12-month period 
and was approved by the EPSB in March 1995. 



What have been the effects of the program on stakeholders? 

Although not validated by research, positive effects appear to be the continuous 
support of KTIP and KPIP among teachers, school administrators, college faculty, and new 
teachers themselves. District teachers and administrators involved on EPSB committees have 
remarked to staff and other members upon their new understanding of, and support for, board 
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priorities and actions. Until their service on these committees, most had known very little 
about the EPSB. 



Lack of congruence between the streamlined certification areas and the Program of 
Studies has resulted in confusion and bottlenecks in districts and among certification staff in 
OTEC, as districts struggling to fill positions contact OTEC staff on a case-by-case basis. 

Preparation programs in colleges and universities report confusion regarding timelines 
for required program changes and for accreditation requirements. Since current staff 
vacancies within OTEC have resulted in an unfilled position for the director of teacher 
education, who is responsible for program approval and accreditation, current staff in that 
division continue to be overloaded and communication with the 26 colleges and universities 
continues to lag. 

The Kentucky Education Association is seeing its legal costs escalate. As district 
administrators learn to work with the mandated reporting system and as the EPSB improves 
and refines its revocation procedures, the number of certified personnel facing revocation 
hearings has steadily increased. 



What are the attitudes and perceptions of stakeholders toward the program? 

Although the EPSB has been in operation since November 1991, there appears to be 
widespread lack of awareness about the board among the general public, many faculty, and 
most school district personnel. Although the EPSB has two members who represent 
professional education and who might reasonably expect to hear from faculty and 
administrators concerning EPSB actions and direction, only 12 of the 26 colleges and 
universities at the time this paper was being prepared were on the KACTE e-mail list 
(coordinated through UK and EKU) to receive board minutes and agendas. Staff report that a 
few institutions (Asbury, Campbellsville, EKU, Murray, NKU, UK, WKU) are regularly 
represented at board meetings (UL and Kentucky Wesleyan currently have a faculty 
member/administrator on the board). 

Within the KDE, the EPSB is only one of many KERA strands, often viewed as a 
very minor component because its impact on the public schools is seen as less direct than are 
other KERA programs; in addition, the board’s unusual administrative relationship in the 
KDE and its reporting structure through OTEC are generally not understood. For their part, 
OTEC staff hold the perception that they are second-class citizens within the KDE, and 
believe that board needs have a low priority. The KDE’s decision to move OTEC to a 
different building in another area of Frankfort during the fall of 1995 has reinforced OTEC 
staffs perception. 

The complexity of the board’s governance issue and the resulting confusion among 
OTEC staff and board members has contributed to the lack of information about the EPSB 
among stakeholders and the general public. Recognizing there is a communication gap 
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between the EPSB and education stakeholders, the board is now issuing a newsletter 
following significant decisions and actions; the newsletter is sent to school districts, 
professional organizations, and preparing institutions. The EPSB is also maintaining and 
publishing information statistics each month, and has initiated an internal news report to keep 
board members and OTEC staff in touch with each other. It is yet too early to discern 
whether the communication gap is closing. 



What otner questions have been addressed? 

The extent to which the New Teacher Standards for Preparation and Certification align 
with KTIP instruments and procedures was addressed in 1993-94, when the board approved a 
revision of the KTIP observation instrument, scoring rubric, and portfolio requirements. The 
board has decided to continue the data collection in relation to these issues during the 1995- 
96 school year. 

The working relationship between the EPSB and the Commissioner of Education has 
been specified in a formal memorandum of agreement. The memorandum was first proposed 
by the board in July 1991 to ensure a foundation for efficient decision-making; while board 
members recognized the need for such a document early in their existence, they were unable 
to accomplish this goal until three years later. 



Other research? 



Studies about professional standards boards prior to 1995 were reviewed and discussed 
in the December 1994 Review of Research on KERA. 

The Appalachia Educational Laboratory commissioned a four-state study on 
professional standards boards (Kentucky, Tennessee, West Virginia, Virginia) for purposes of 
identifying ways for the profession to have regulatory and statutory impact. The papers were 
presented in mid-October 1995 (for the Kentucky portion, see Nelli, Leib & Martray, 1995). 
At the time of the writing of this review, the titles of other papers were unavailable, although 
the authors’ names are: Jane Applegate, West Virginia; Gary Ellerman, Virginia; Connie 
Smith, Tennessee). Presumably the proceedings will become available in the near future. 

The Connecticut Education Association (1995) identified and summarized the 
characteristics of autonomous state teacher professional standards b "•'trds in eleven states, 
including Kentucky, with its semi-autonomous status. 



What research is in progress in this area of KERA? , , ; 

The Cheyney Project Evaluation (August 1994), described in detail in the December 
1994 Review of Research on KERA, raised important questions about assessment, including 
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the need for experts in assessment over a long time frame to provide assurance regarding the 
validity, reliability, and generalizability of Kentucky assessments. The board is in the process 
of establishing a Third Party Critical Colleague Panel, which will meet two or three times 
annually and will be composed of out-of-state assessment experts. The panel will review all 
EPSB assessment development in Kentucky, and the use of assessments in certification. The 
panel is expected to have its first meeting in fall 1995. 

OTEC staff are continuing their work with two national projects of critical importance 
to the profession: INTASC (performance assessments) with data collection to continue during 
the 1995-96 school year, and recreating the NTE in collaboration with ETS, to result in 
validated tests to be known as Praxis II. Validation panels of subject area teachers are 
currently examining subject-specific test questions. The expected completion date of the ETS 
project is September 1, 1996, with passing scores set in Kentucky for every subject area 
except visually impaired and hearing impaired (and possibly driver education). 

OTEC staff continue to coordinate field testing in other areas of teacher standards and 
assessment, specifically on-demand tasks and portfolio assessment. Collection of assessment 
data appears to follow pertinent recommendations of Kifer and Guskey (1994) so that for the 
first time, linkages will be possible for researchers interested in assessment results across 
areas: on-demand tasks, portfolio reviews, and test results. 

A KTIP pilot project, approved by KTIAC but not yet reported to the EPSB, is 
currently studying the use of streamlined training processes and technology to update 
classroom observers in relation to the revised observation instrument, new performance 
standards, and KTIP expectations for observers. 



Which questions or issues need further research? 

The December 1994 Review of Research on KERA suggested two major areas needing 
further research: the efficiency and effectiveness of the EPSB as a governing body, and the 
validity and reliability of performance tasks for use in professional licensure. Those research 
needs remain. 

With the 1995 AEL-sponsored study and the Connecticut Education Association 
(1995) summary, a start could be made on researching the board’s autonomy and 
effectiveness. Nelli, Leib, and Martray (1995) note that the EPSB’s organizational placement 
under the KDE has minimized operating costs, but having an executive secretary reporting 
both to the board and to the commissioner of education has been almost untenable. After 
years of priority conflicts and miscommunication between the commissioner and the beard, 
the memorandum of agreement between the two parties has somewhat eased the tension. 

Even so, the board’s staff remain within the KDE (although OTEC will soon be 
physically removed from the KDE), which means that the memorandum ol agreement 
depends on the good faith efforts of both parties. The issue to be addressed is how the board 
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can be staffed and supported so as to ensure its autonomy without necessitating major outlays 
of state funds at a time of diminishing budgets. 

A second point raised in the Kentucky portion of the AEL papers is the preparedness 
of board members to serve on a statewide governance body. The legislature mandated the 
length of term for EPSB members with rotating appointments and statewide representation to 
ensure diversity and widespread input. These rotations, however, have resulted in a lack of 
continuity and an inconsistency in policy decisions from one year to the next. With the 
board’s work progressing at breakneck speed, particularly in assessments, revocation 
procedures, and licensure decisions, the issue of preparation and training for board members 
becomes critical. 

As new members join the board, it takes time for them to learn board business and 
procedures; until they do, the EPSB cannot continue functioning as a cohesive unit rather than 
as a body representing the distinct and special interests of individual board members. The 
issue to be ~ >nsidered in Kentucky is how board members can best be prepared to represent 
their constituents in both diversity of membership and consistency of action. 

As noted in the December 1994 Review of Research on KERA, the participation in 
teacher education decisions of faculty in arts and sciences remains problematic, and is a 
recurrent theme in teacher education. Research is desperately needed to identify processes 
existing within the state and within institutions that effectively facilitate the involvement of 
arts and sciences faculty in the preparation of teachers and other school personnel. 

Related to the involvement of arts and sciences faculty in professional education is 
another crucial issue, that of subject matter knowledge and the importance of integrating 
specific subject matter knowledge with pedagogy in the development of teacher assessments. 
The December 1994 Review of Research on KERA points out that such is critical to the 
original goals of the assessment projects in Kentucky, and is equally pertinent to candidates in 
alternative certification programs. 

Formal research and reported results are essential if the education stakeholders— and 
equally important, members of the Education Professional Standards Board-are to become 
aware of, and understand, the impact of the various policy decisions and initiatives of the 
board. The board should make clear to potential researchers the availability of the masses of 
data collected by OTEC during pervious years, and particularly data from 1994-95 and 1995- 
96, and should specify the need for research results to help guide future EPSB decisions. 
What, for example, has been the impact of board decisions made in streamlining certification, 
program approval/accreditation procedures, minority recruitment, KTIP? 

In similar manner, the board should welcome serious research regarding its own 
operational needs: What strategies and policies might succeed in enabling the EPSB to 
function in an autonomous manner? How can the board prepare new members for immediate 
and effective functioning? How can the board ensure that communications about its plans 
and actions reach stakeholders— and are attended to ay these stakeholders? 
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ASSESSMENT AND ACCOUNTABILITY 



Joseph M. Petrosko 
University of Louisville 



What does the law or regulation mandate? 

Section 158.6453 of the Kentucky Revised Statutes (KRS) deals with assessment of 
achievement goals, the development of a statewide assessment program, and the publication 
of an annual performance report by school districts. This section of the law requires that, no 
later than the 1995-96 school year, the State Board for Elementary and Secondary Education 
will develop a statewide assessment system, including performance-based student testing. The 
purpose of the system is to measure the success of each school in achieving the learning goals 
and learner outcomes described in the law. An interim testing program was to be 
administered during the 1991-92 school year requiring the assessment of student skills in 
reading, mathematics, writing, science, arid soc ; al studies in grades 4, 8, and 12. In order to 
provide baseline data for determining school success in subsequent years, test scores were to 
be used, along with four non-cognitive factors (rates of attendance, retention, drop-out, and 
successful transition to adulthood). Tests were to have been constructed to allow the state to 
make national comparisons, and to be similar to those used by the National Assessment of 
Educational Progress (NAEP). In addition to the accountability testing in grades 4, 8, and 12, 
the State Board was given the responsibility of assisting local school districts in developing 
•and using continuous assessment strategies needed to assure student progress. 

Regarding accountability. Sections 158.6455 and 158.782 of the Kentucky Revised 
Statutes (KRS) deal with determination of and rewards to successful schools, the development 
of school improvement plans for schools not meeting goals, a description of "schools in 
crisis," appeals of performance judgments, and a description of "distinguished educators" who 
will be assigned to help schools implement improvement plans. 

Based on performance test results and other data collected in 1991-92, each school in 
the state was to have been assigned a number representing its level of success in meeting 
educational goals— a baseline value of performance. Then a threshold level of school 
improvement was to have been calculated from the baseline. A comparison was to have been 
made between the threshold value and the performance for the school from a two-year period 
(the average of data from 1992-93 and from 1993-94). The comparison was to have been 
used to place the school into one of five categories. 

1. Eligible for Monetary Rewards: if the school scores at least a 1% gain over the 

threshold and at least 10% of the students at the lowest level of achievement 

("novice") move to a higher level. 

2. Successful: if the school scores at its threshold or exceeds it by less than one point. 
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3. Not Meeting Threshold or Improving: if the school scores at or above its baseline but 
below its threshold (school improvement plan must be developed). 



4. In Decline: if the school scores from 0 to 5% below its baseline (school assigned a 
"Kentucky distinguished educator" to assist in developing an improvement plan). 

5. In Crisis: if the school scores more than 5% below its baseline (school assigned a 
"Kentucky distinguished educator" to evaluate the school program and personnel.) (In 
addition, all certified staff are placed on probation and parents will be informed of 
their right to transfer their children to successful schools.) 



In 1994, KERA was changed by the General Assembly to postpone the sanctions 
associated with the "In Crisis" category. Thus, schools that would have met the criterion for 
that category in 1995 were instead categorized "In Decline." 



Review of Research to date. 

To what extent has the program been implemented as intended by KERA? 

Because of the complexity of assessment and accountability, two dimensions of 
implementation will be examined. First, evidence will be analyzed as to whether assessment 
and accountability systems with the features intended by the authors of the reform law were 
put into place. Regarding this dimension, "basic implementation" data were examined to 
determine whether the legislature’s intentions for this area of KERA were addressed, at least 
in some minimal way. Secondly, research studies regarding the quality of implementation 
were reviewed— i.e., whether the systems that came into existence met recognized standards of 
acceptability: a dimension called "implementation quality." 

Basic Implementation 

According to the mandate of the legislature, a student assessment system was to have 
been established that possessed several key elements. The assessment was to have emphasized 
"authentic tasks" relevant to the goals of KERA. In addition, an interim assessment system 
was to have been administered in 1991-1992 to provide baseline data for determining 
successful schools and to allow districts to publish data in local newspapers. Further, 
accountability indices and threshold values for Kentucky schools were to have been 
calculated. The assessments were to have been designed to allow Kenucky students to be 
compared with nationally representative groups of students. 

Data on basic implementation were obtainable from the reports of the contractor that 
designed the assessment system to state specifications: Advanced Systems in Measurement 
and Evaluation, Inc., of Dover, New Hampshire (hereafter called ASME). In January 1993, 
ASME issued a 94-page technical report to the Kentucky Department of Education that 
described the development and implementation of the system in its first years of operation 
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(Advanced Systems in Measurement and Evaluation, Inc., 1993a). In addition, a set of 15 
appendices to the main report was issued, which included copies of test items and various test 
score conversion tables (Advanced Systems in Measurement and Evaluation, Inc., 1993b). 

In August 1994, an 89 page report was issued containing 12 chapters dealing with 
assessment activities that took place in the 1992-1993 school year. A volume of 15 
appendices was published with the report (Advanced Systems in Measurement and Evaluation, 
Inc., 1994b). 

Finally, in July 1995, a 16 chapter manual was published (Kentucky Department of 
Education, 1995) which summarized all data.from the first "accountability cycle"— 1991-1992 
through 1993-1994. The 242 page manual repeats much of the data in the two ASME 
reports, but also contains previously unpublished information. The KDE (1995) manual was 
written by employees of both ASME and the state department of education. Taken together, 
these documents written by ASME and KDE contain information on topics such as test 
construction procedures, reliability, validity, and the scoring and rescoring of writing profiles. 

From the two ASME reports and the KDE report, evidence is available to answer 
whether basic implementation took place of the assessment and accountability systems 
envisioned by the legislature. For example, the first 12 chapters of KDE (1995) reveal that, 
indeed, a performance-based student assessment system was developed. Statewide assessment 
data were collected from students in 1991-92, 1992-93, and 1993-94. The Kentucky 
Instructional Results Information System (KIRIS) was designed to provide baseline data and 
to provide data for an accountability index to determine successful schools. As required in the 
law, KIRIS gives districts the data on the performance of schools that can be published in 
local newspapers. Furthermore, KIRIS items were developed, in part, using test frameworks 
derived from the National Assessment of Educational Progress (NAEP). This responds to the 
mandate of KERA that state assessment have "NAEP-like" tests items. As reported in KDE 
(1995), "KDE has defined NAEP-like to mean having substantially similar content 
frameworks and to be statistically linked for the purpose of inferring national norms" (p. 13). 

The accountability index calculated for each school reflects several sources of data. 
Students in grades 4, 8, 12 and other grades were tested in a variety of ways in the 1991- 
1992 and subsequent school years. In testing called "transitional," students received multiple- 
choice and open-ended tests, and responded to writing prompts. Tests had some common 
items and some matrix-sampled items. The open-ended items allowed students to construct 
responses to questions in their own words. Such items were scored by trained graders, and 
were consistent with the philosophy of performance assessment. Students were also tested 
with "performance events." These involved having testers come to schools and examine 
students, who worked in small groups, with open-ended questions, problems, or tasks (in 
mathematics, science, and social studies). Finally, writing and mathematics portfolios were 
collected from students. The great variety of assessment formats in KIRIS were designed to 
meet the legislature’s requirement for alternatives to a purely multiple-choice test. 
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The reports from ASME and KDE provide important documentation, but are not the 
only sources for understanding this area of KERA. Two large-scale evaluations of the 
Kentucky assessment and accountability systems were published in 1995, and both have a 
bearing on the question of basic implementation. The first to issue a report (The Evaluation 
Center, 1995) was commissioned by the non-profit Kentucky Institute for Education Research 
(KIER), as part of its overall mission to conduct an in-depth evaluation of the impact of 
KERA on students, schools and teachers. The study was performed by The Evaluation Center 
of Western Michigan University. A second evaluation of Kentucky’s assessment (Hambleton, 
Jaeger, Koretz, Linn, Millman, & Phillips, 1995) was sponsored by the Office of Education 
Accountability, an arm of the state legislature charged with monitoring the implementation of 
the education reform law. 

The authors of the KIER-sponsored evaluation concluded that, "overall, KIRIS is 
consistent with the Kentucky Education Reform Act" (The Evaluation Center, 1995, p. 5). 

This conclusion was largely based on document analysis, surveys of educators, and focus 
groups with stakeholders. The evaluators stated that KIRIS met the legislature’s intention of 
constructing a performance-based high stakes assessment, with potential links to national 
norms via the NAEP assessments. 

The OEA-sponsored study (Hambleton, et al., 1995) mostly focused on psychometric 
issues, and will be discussed in the next section of this review. Generally speaking, the 
authors were critical of many aspects of KIRIS. However, it would seem that Hambleton, et 
al., at least on a tacit basis, accepted the Kentucky assessment and accountability systems as 
having the basic features that were intended by those who wrote the law, e.g., a largely 
performance-based assessment system to be used for school accountability. For example, the 
authors offer a considerable number of recommendations which they hope "will be valuable to 
the Department of Education as it sets out on the second accountability cycle" (p. 9-10). The 
authors called for changes in such things as the calculation of accountability scores and 
methods of test equating. They do not recommend the wholesale scrapping and replacement 
of the present system, as would be expected if the authors believed KIRIS was totally at 
variance with the intentions of the law. 

Summary. Data from three sources were examined: the testing company that 
developed KIRIS and two independent evaluations. Based on examination of these sources, 
the author believes that basic implementation occurred of the assessment and accountability 
systems envisioned by the authors of the school reform law. 

Implementation Quality 

The issue of implementation quality is much more problematic and debatable than the 
issue of basic implementation. This section will contain an examination of the evidence 
related to whether the assessment and accountability systems met generally recognized 
standards of acceptability. For the most part, the evidence considered in this section will deal 
with psychometric issues, such as measurement reliability and validity (as these are 
traditionally defined). 
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Advanced Systems in Measurement and Evaluation and the Kentucky Department of 
Education. The most comprehensive documentation of KIRIS and its accompanying 
accountability system is contained in the volumes produced by ASME and KDE. Because the 
latest of these volumes (KDE, 1995) was designed to be the most complete in its coverage 
(viz., reporting on data from all years of the system), it will be extensively reviewed. This 
will be followed by reviews of other research sources bearing on how well KIRIS has been 
implemented. 

The first section of KDE (1995) contains two chapters that give the context of 
assessment and accountability systems. In the first chapter, the authors state that educational 
change takes many years to have a noticeable impact: "a sufficient number of students must 
experience the changed curriculum for enough time to show an effect" (KDE, 1995, p. 2). 

Thus, they believe that a limitation of documenting the system after four and a half years is 
that not enough time has passed to see many effects. 

The second chapter provides background for KIRIS, including the various sources of 
educational goals for Kentucky youth. KERA listed seven "capacities” required of students in 
public education. To further explain what was required, in 1991 the State Board of Education 
adopted 75 statements that more specifically defined what was expected of students. Then in 
1994, the list was pared to 57 "academic expectations" that make up the current description of 
"what Kentucky students should be able to know and do when they graduate from high 
school" (KDE, 1995, p. 5). For example, under the general goal of communication skills is 
this academic expectation: "Students make sense of the variety of things that they read" 

(KDE, 1995, p. 7). 

The authors describe the accountability system in Chapter 3. Each school is measured 
on cognitive and non-cognitive outcomes. Cognitive data are derived from on-demand tests 
(i.e., multiple-choice tests, open-ended tests, essay tests requiring student responses to writing 
prompts), performance events (groups of students given problem-solving tasks), and portfolios 
(in writing and mathematics). 

In addition to the cognitive measures described above, non-cognitive variables were 
measured at all schools. From primary to high school, two measures were collected. These 
were attendance rate and retention rate. Attendance refers to percentage of enrolled students 
who are showing up for school. Retention means percentage of students held back a year (a 
low percentage is better than a high percentage). For high schools, two additional non- 
cognitive measures were collected, dropout rate and "transition" rate. Dropout rate refers to 
the number of youngsters who leave school before getting a diploma. Transition rate means 
successful transition to adult life— the percentage of twelfth graders who go to a job, the 
military, or further education rather than "graduating" to the unemployment rolls. 

Using results from both the cognitive and non-cognitive measures, and going through a 
series of arithmetic calculations, each school was assigned an accountability index. Chapter 3 
of KDE (1995) briefly describes the index, but a better example is in Appendix F of the 1993 
ASME technical report appendices (ASME, 1993b). The latter shows what the source data 
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would look like in calculating the index from a hypothetical high school. In the example, 52 
numbers from a data matrix (e.g., percentage of students achieving a particular level in 
science performance events, percentage of students dropping out) were weighted in various 
ways and subjected to additional calculations—eventually leading to one number - , an 
accountability index. For the data in the example, the index for a hypothetical high school 
was shown to be 39.5. 

The index is used to compute a threshold, that is, a target for future performance. The 
process assumes that the theoretical ideal accountability index for a school is 100. In the 
hypothetical data presented in Appendix F of the 1993 ASME technical report, the index of 
39.5 is 60.5 different than the ideal (100 - 39.5 = 60.5). To set the threshold for the school, 
10% of the discrepancy number is calculated: 10% of 60.5 is approximately 6.1 accountability 
points. This is added to the school’s accountability index, 6.1 + 39.5, and the result, 45.6, is 
the threshold value, which is a target for performance when the school is measured and tested 
over the next two years. In other words, when cognitive and non-cognitive measures are 
averaged over the 1992-93 and 1993-94 school years, the school must have a 45.6 
accountability index to meet its threshold value. 

Section II of the KDE (1995) report contains four chapters on test development and 
item analysis. This starts, in Chapter 3, with information on the test specifications used in 
KIRIS. Items were designed to fit several different frameworks and sets of guidelines. These 
included two sources discussed in the previous chapter: (a) Kentucky’s learning goals, (b) the 
5 / learner expectations, as well as, (c) a state curriculum framework called Transformations , 
and (d) NAEP frameworks. The chapter contains 35 tables giving basic information such as 
which of the 57 expectations were covered at which grade level (4, 8, or 12) and in which 
year of testing (1991-92, 1992-93, or 1993-94). Also included in the chapter were examples 
of the types of assessment items (e.g., multiple-choice, open-ended) used in KIRIS. 

Chapter 4 of KDE (1995) describes the test development process. It consists of 19 
steps, and includes significant involvement from two sets of groups. The first are content 
advisory committees of Kentucky educators, with separate committees in seven different 
areas, e.g., mathematics, reading. These persons advise on a variety of issues, such as which 
items to include among the "common" items in KIRIS (selected from the previous year’s 
matrix sampled items) and what concepts need to be covered by new items. The second set 
of persons heavily involved in test development are the ASME technical staff, who perform 
item analyses and carry out field tests of new items. The items included in KIRIS reflect the 
educational judgements of state educators and the technical judgments of staff using 
psychometric criteria like item/total correlations and mean difficulty level of items. 

A number of different tasks were involved in setting performance standards in KIRIS. 
As described in chapter 5 of KDE (1995) all of these related to the general issue of 
determining what levels of performance constitute a high or low level of achievement and 
thus could be used as evidence that a school is "successful" or not. 
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Considerable effort was expended in selecting how many categories of performance 
would be used to characterize students and schools and what to name the categories. After 
some debate, four categories of student performance were designated: a) novice, b) apprentice, 
c) proficient, or d) distinguished. Novice means low achievement and apprentice somewhat 
higher, but both indicate levels of performance that need improvement. The goal of KERA is 
to move all children to the proficient level in all subjects. The last category, distinguished, 
was reserved for students displaying extraordinary performance in an academic area. 

An additional task was to develop standards for open-ended test items. Standard- 
setting committees of Kentucky educators were employed to examine the items and determine 
what kind of response would merit a particular numerical score, ranging from 0 (zer^j "no 
response or irrelevant response" to 4 "complete, insightful, strongly supported response, 
demonstrating in-depth understanding of relevant concepts and processes" (KDE, 1995, p. 68). 
Then, judgments were made as to what total score on a set of open-ended items constituted 
each of the student performance categories (e.g., novice, apprentice). Also included in 
chapter 5 were sections related to standard setting for writing portfolios, mathematics 
portfolios, and alternate portfolios (used with the severely disabled). 

Chapter 6 of KDE (1995) provides 21 tables that summarize analyses of open-response 
and multiple-choice items. The tables give data on four subject areas (mathematics, reading, 
science, and social studies) and three grades (4, 8 and 12). One set of tables show the 
distribution of open-ended item scores: how many items had an average score of 0.00 to .40, 
.41 to .80, up to a maximum of 3.61 to 4.00. Since a score on an open ended item could 
range from 0 to 4, a low average item score meant a difficult item and a high average score 
meant an easy item. For all subject areas and grades, most average item scores were between 
1.21 and 2.00. 

Also shown in Table 6 were bi-serial correlations— which measured the relationship 
between an item score and the performance category of the student (e.g., novice, apprentice). 
It was found that items differentiated students who were novice from those who were above 
novice "reasonably well" (KDE, 1995, p. 83). (If one assumes that novice, apprentice, 
proficient and distinguished are Levels 1, 2, 3 and 4, respectively, this would be 
distinguishing Level 1 from Levels 2, 3 and 4.) However, items did not differentiate as well 
those students who were apprentice or below from those who where proficient and above. 
(This would be distinguishing Levels 1 and 2 from Levels 3 and 4.) 

Section III contains six chapters on administration, scoring, and reporting of the data 
from the KIRIS and school accountability systems. Chapter 7 describes test administration 
procedures, and includes information on rules for determining students for whom a school is 
accountable (i.e., when in the school year a student had to be enrolled in a school for that 
student’s score to be counted in the school’s accountability index). Data are also included on 
the numbers of students tested. Many students were tested with KIRIS: approximately 
52,000 fourth graders, 53,000 eighth graders, and 37,000 twelfth graders in each of the 
accountability years 1991-92, 1992-93, and 1993-94. 
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Chapter 8 deals with scoring, which involved trained raters and graders, because so 
much of KIRIS involved assessment with open-response items, portfolios, and other non- 
multiple choice formats. The chapter provides information on scoring of open-response test 
items, including the number of scorers hired, their qualifications, and scoring reliability 
statistics. Statistics are presented on the degree of agreement between persons scoring open- 
ended items and training leaders. For the 2% of student papers compared in this way, the 
percentage of exact agreement ranged from 81.1% in grade 12 science to 93.4% in grade 8 
math. For three grade levels and four content areas and, the average percentage of exact 
agreement was 85.1% (KDE, 1995, p. 122). Also reported for open-ended tests were, for 
individual student data, correlations between original scores and rescores of the items. Out of 
12 correlations reported (three grade levels by four content areas), one was .89 and the 1 1 
others were .90 or above (KDE, 1995, p. 122). 

Chapter 9 of KDE ( 1995) summarizes the steps used to equate KIRIS items from year 
to year. Equating refers to the mathematical procedures used to adjust test scores and make 
them comparable from one year to the next. Test items, even those having a similar content, 
differ from one another in difficulty and discriminability (i.e., their ability to distinguish 
between high-achieving and low-achieving students). Without equating test items, there 
would be measurement error added to the data when comparing scores from one year to 
another. 

Equating in KIRIS involved applying a two-parameter item response theory (IRT) 
model (one parameter for difficulty, one for discrimination). It was a technically complex 
process: to make a long and complicated story short, open response test data for 1993-94 
were adjusted in light of information from 1992-93 in order to maintain equivalence. The 
basis for the equating were matrix sampled items. Students took fewer of these than common 
items, but for a number of reasons (see KDE, 1995, p. 124) matrix items were thought to be 
the best choice. There were no attempts to equate data from 1993-94 to 1991-92. However, 
KDE deemed it sufficient that the 1992-93 data had been equated to the 1991-92 data. 

Chapter 9 contains adjustment coefficients used to equate the 1993-94 items to the 
1992-93 items. It also contains results of studies undertaken to verify the technical decisions 
made in equating. Finally, the chapter has information on the equating of data derived from 
performance events. 

Chapter 10 of KDE (1995) is very brief and describes the method of reporting to 
schools and districts. A large volume of information was transmitted to schools, including 
individual student reports, summaries of cognitive data by school, and detailed information on 
multiple-choice and open response test items. 

In Chapter 1 1 of KDE (1995) the statewide accountability scores for the first 
"accountability cycle" are presented. It can be argued that these data are the most significant 
pieces of information presented in the entire report. 
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Below are some key statistics. 



Kentucky Average Accountability Index Scores 
1992-94 





Baseline 


Average Index 


Gain 


Percent Gain 


Grade 4 


33.9 


41.5 


7.6 


22 


Grade 8 


37.4 


42.4 


5.0 


13 


Grade 12 


39.3 


‘42.8 


3.5 


9 



Source: Table 11.1 (KDE, 1995, p. 147). 



The results above are average accountability scores, which reflected both cognitive and 
non-cognitive data. Results revealed gains at all levels, and that, the lower the grade level, 
the greater the gain. Similar results occurred when examining various subsets of the data (e.g., 
only cognitive data). The authors concluded that "statewide performance showed impressive 
gains" (KDE, 1995, p. 147), and that gains were greatest in primary school, followed by 
middle grades, followed by high school. 

# 

Chapter 12 of KDE (1995) provides information on monitoring the scoring accuracy 
and reliability of writing portfolios. The chapter summarizes seven studies done since 1992, 
all aimed at improving scorer reliability. On the basis of these studies, the authors concluded 
that "many schools have made dramatic improvements in both student performance and 
scoring accuracy" (KDE, 1995, p. 152). It was claimed that local teachers and review scorers 
(persons rescoring a portfolio) have shown increasingly high levels of agreement. Further, 
inter-rater reliabilities of .80 are common, as calculated on data from review scorers. 

Auditing procedures have allowed KDE to identify schools with suspiciously high portfolio 
scores and to attempt remedies for such inaccurately high scores. Overall, the authors 
concluded that there have been "significant improvements in research and quality control 
procedures since 1991" (KDE, 1995, p. 152). 

While data on scoring reliability were positive, some aspects of portfolio scoring were 
cause for concern. For example, portfolio scores for local teachers were adjusted on the basis 
of what scores would have been had the portfolios been rated by review scorers (i.e., persons 
unconnected with the students who produced the portfolios). This adjustment procedure is 
called "recasting." In commenting on the results of this process (shown in Table 12-29), the 
authors concluded that "recasting lowers scores in every case-that is, teachers have scored 
portfolios too high and continue to do so" (KDE, 1995, p. 193). 

In summary, research on writing portfolios indicates that progress has been made in 
scoring accuracy (i.e., the degree of agreement among scorers as to whether a portfolio 
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manifests low or high achievement). However, there is a tendency for local teachers to mark 
their students’ portfolios higher than review scorers (i.e., outside readers). 

The last section of KDE (1995) is section IV, titled Generalizability, which contains 
data on reliability and validity. For example Chapter 13 provides data on the reliability of 
KIRIS accountability indices. Included in the chapter are internal consistency reliability 
coefficients (Cronbach’s alpha) from individual open-response student tests. Table 13-1 (p. 
197) shows three sets of coefficients: 1991-92, 1992-93, and 1993-94. For each set, there 
were 15 coefficients (three grades— 4, 8, and 12— by five categories— reading, mathematics, 
science, social science, and composite of all subjects). The range of alpha coefficients 
(ignoring the composite category) was as follows: (a) 1991-92— .58 to .79, (b) 1992-93-.65 

to .85, and (c) 1993-94— .66 to .85. The following conclusion was reached: "Although the 
Department of Education advises against making student-level decisions based on individual 
test scores alone, open-response test reliabilities compare favorably with other tests used to 
make student-level decisions" (KDE, 1995, p. 196). The authors cite as examples the 
American College Testing (ACT) placement test and the subject tests of the Comprehensive 
Tests of Basic Skills (CTBS). 

Chapter 13 of KDE (1995) provides data on the reliability of the KIRIS accountability 
indices. The authors presented data derived from a psychometric approach called 
generalizability theory. Generalizability coefficients (akin to reliability coefficients) were 
calculated for school-level data at grade 12 (presumably from the cognitive index part of the 
school accountability index) . 

In calculating generalizability coefficients, a number of different assumptions were 
made. One of these related to whether students should be considered a "fixed" or "random" 
source of error. (This terminology in generalizability theory is derived from analysis of 
variance.) The same issue came up with test items; they could be considered fixe^ or 
random. The authors of KDE (1995) calculated all possible combinations of assumptions for 
grade 12 data in 1992-93 and 1993-94. Data did not change much based on these 
assumptions. For example, in 1993-94, generalizability coefficients ranged from .88 (students 
and tasks both considered random) to .92 (students and tasks both considered fixed). Data 
were very similar in 1992-93. The authors concluded that "these data indicate an adequate 
level of reliability in each year for grade twelve schools as small as 48 students" (KDE, 1995, 
p. 198). 

Chapter 14 of KDE (1995) contains a description of validity-related evidence on 
KIRIS. Traditional types of validity (content-related, criterion-related and construct-related) 
are discussed. However, the authors argue that consequential validity, the effects of KIRIS on 
instructional practices, constitutes an important additional criterion forjudging Kentucky’s 
assessment. One aim of KIRIS is to improve classroom instructional practices of teachers. 
This means greater involvement of teachers in assessment, primarily in the development and 
scoring of portfolios. If high-teacher-involvement assessment methods like portfolios were 
sufficiently reliable, they would form the major component of KIRIS. Portfolios are only one 
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part of assessment because "they do not yet have reliability at the desired level" (KDE, 1995, 

p. 200). 

Regarding content-related evidence for KIRIS, the authors argue that this is shown by 
the links to Kentucky’s academic expectations and other item frameworks (in chapter 3 of 
KDE, 1995). Regarding construct-related evidence, the chapter contains tables showing inter- 
correlations among different kinds of KIRIS scores. For example, data at the school level 
show correlations among 14 KIRIS scores: multiple-choice scores in four subject areas, open- 
response scores in four subject areas, on-demand writing, two portfolio scores, and three 
performance event scores. All correlations were positive. However, correlations between 
performance events and other measures (e.g. ppen-response) tended to be low. Also, there was 
some evidence of a "method effect." This means that the method of data collection tended to 
affect the strength of relationships among measures. For example, examining school-level 
correlations at grade 8, scores on multiple-choice reading and multiple-choice math correlated 
.78. However, multiple-choice reading and open-response reading correlated .63, and 
multiple-choice math and open-response math correlated .56. In other words, there was a 
higher correlation between two different subjects (math and reading) measured with the same 
method (multiple-choice) than there was with the same subject (e.g., reading) measured with 
two different methods (multiple-choice and open-response). The authors argued that such 
method effects are to be expected, given the fact that different testing methods require 
different abilities from students. 

Concurrent validity statistics were reported for KIRIS instruments and several other 
tests. Correlations between scores on the Comprehensive Tests of Basic Skills (CTBS) scores 
and 1991-92 open-response questions were reported. There were 16 correlations in all, 
covering reading and mathematics at grades 2 through 9-11. Sample sizes ranged from 3,846 
(grade 6 mathematics) to 31,768 (grade 4 reading). Each correlation involved normal curve 
equivalent (NCE) scores from the CTBS and raw scores from open-response questions. The 
16 correlations ranged between .47 and .65. It was concluded that "given the limited number 
of questions on the KIRIS tests, these data show a modest correlation between the two tests" 
(KDE, 1995, p. 211). 

Correlations were reported between KIRIS open-ended mathematics scores and 
National Assessment of Educational Progress (NAEP) math scores. Data came from samples 
of students: 2,585 at grade 4, 2,702 at grade 8, and 997 at grade 12. KIRIS math and NAEP 
math scores were correlated .74, .78, and .79 at grades 4, 8, and 12 respectively. Finally,. 
ACT college entrance scores and KIRIS scores were correlated for 20,610 students measured 
in 1993-94. Correlations were .47 for reading, .67 for mathematics, and .46 for science. 

Chapter 15 of KDE (1995) presents data on consequential validity. This will be 
discussed later in this review, in the section dealing with the effects of the assessment and 
accountability systems. The last chapter of the KDE technical manual, chapter 16, deals with 
gender and ethnic differences, a topic that is related to the issue of implementation quality. 
Data on open-response items and multiple-choice items were compared for these groups: (a) 
males and females, and (b) majority and minority students. The analysis revealed that, on 
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average, girls did better than boys on both open-response items and multiple-choice items. 
However, the magnitude of the gender difference was greater for open-response items than 
multiple-choice. The analysis also revealed that, on average, white students did better than 
black students on both open-response items and multiple-choice items. However, the racial 
gap was less on open-response items than multiple-choice. 

Office of Education Accountability Measurement Panel. KDE (1995) contains a 
wealth of information on implementation quality of KIRIS. However, it is not the only 
source of information on this issue. Two major evaluations of the KIRIS and accountability 
systems were published in 1995 that have important implications for the future of KERA. The 
first of these to be reviewed was the evaluation sponsored by the Office of Education 
Accountability (OEA) (Hambleton, et al., 1995). 

The evaluation panel of six measurement experts was empowered to address the 
following questions: "Is the measurement quality of KIRIS sufficient to support the intended 
uses of KIRIS results . . .? And, to the extent that shortcomings in KIRIS are identified, what 
changes need to be made to improve the . . . system?" (Hambleton, et al., 1995, p. 1). In 
addressing these questions, the panel spent most of its time examining technical data obtained 
from the Kentucky Department of Education (KDE). These major aspects of KIRIS were 
examined: (a) test development procedures, (b) reliability of accountability and assessment 
scores, (c) reliability and validity of portfolio scores, (d) comparability of scores across years 
(i.e., equating methods), (e) defensibility of performance standards, (f) quality of reporting of 
assessment and accountability results, and (g) degree of improvement in Kentucky education 
since the implementation of KIRIS. An appendix to the report, authored by one panel 
member who is an attorney, dealt exclusively with legal issues raised by KIRIS. Work was 
performed between December 1994 and June 1995. 

The panel was critical of every aspect of KIRIS that its members examined. As a 
consequence, the evaluator’s major conclusion was that "as KIRIS is currently designed and 
implemented, it is flawed to the extent that ... the present form of KIRIS cannot support its 
accountability and assessment goals and objectives . . . also ... (1) the public is being 
misinformed about the extent to which student achievement has improved statewide; and (2) 
the public is being misled by being given information about the accomplishments of 
individual students that may be inaccurate" (Hambleton, et al., 1995, p. 5). 

Five findings supported this main conclusion. Three of these are sufficiently 
important to merit some discussion. First, it was found that misclassification rates in some 
reward categories were high, making the rewards and sanctions "difficult to defend" 
(Hambleton, et al., 1995, p. 5). The data in question here were not discussed in KDE (1995). 
However, as was stated previously in this review, each school in the state was evaluated in 
terms of its gains in accountability index scores (largely determined by cognitive test data). 
Schools were assigned into one of five categories: eligible for reward, successful, improving, 
in decline, or in crisis. In 1995, the money distributed to Kentucky educators in schools that 
were "eligible for reward" totaled approximately 26 million dollars. 
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The placement of a school in a category was affected by sources of variance such as 
what cohorts of students were tested and the particular assessment tasks that were used in a 
school. By performing computer simulation studies, it was possible to estimate how likely it 
was that category assignment for the school was the "true category" rather than an erroneous 
classification due to sampling error. After examining such studies done by the Department of 
Education, the panel implied that misclassification rates were too high. For example, in 
commenting on one such study, they stated "the odds that a school that would have been 
classified as In Crisis at Grade four truly declined are only two to one. In other words, 1 in 3 
schools that would have been classified as In Crisis in Grade 4 would have actually improved, 
or at least showed no decline" (Hambleton, et al., 1995, p. 3-20). 

% 

Another finding that influenced the panel was a consideration of data from other 
assessments, besides KIRIS, that have been used to measure Kentucky students since the 
inception of KERA. While KIRIS showed substantial gains for Kentucky students, other data 
did not. Probably the most relevant were data from the NAEP trial state assessment (because 
the NAEP approach to measuring reading and the KIRIS approach have similarities). The 
panel compared average reading scores of fourth graders from 1991-92 to 1993-94. The 
standardized change (in z score units) was +0.76 for KIRIS, but -0.03 for NAEP reading. A 
similar lack of gain occurred for ACT test scores over the same time period. As a 
consequence, it was concluded that "taken together, the ACT and NAEP findings are 
sufficient to suggest that gains in KIRIS scores are substantially inflated and provide the 
public with a misleading view of improvements in student performance" (Hambleton, et al., 
1995, p. 8-4). 



A third major finding that affected the panel’s major recommendation was its belief 
that portfolio scores lacked sufficient reliability or validity to be included in the accountability 
system. One of the biggest problems identified was the tendency for teachers to rate the 
portfolios of their own students higher than the scores obtained when outside readers graded 
the same student work. For example, in one set of data, a sample of grade 8 portfolios were 
rescored in the summer. Of the portfolios that teachers rated as Proficient, only 29% were 
scored that way by readers during rescoring— 69% were scored as Apprentice or Novice 
(Hambleton, et al., 1995, p. 4-15). This indicates a large discrepancy between the level of 
writing ability as perceived by the teacher and as perceived by an outside person with no 
previous knowledge of the student or no financial stake in the assessment results for the 
school. 



Two other major findings of the panel related to equating and standard-setting. The 
panel concluded that poor technical practices led to "an accumulation of equating errors which 
make the year to year comparisons of KIRIS results of questionable validity" (Hambleton, et 
al., 1995, p. 6). In addition, it was concluded that flawed procedures were used to set 
standards. For example, standards were based on three test items per subject/grade 
combination— too few to provide stable estimates of student performance. Thus, the panel 
stated that it considers "the performance standards used to classify students as Novice, 
Apprentice, Proficient, and Advanced [i.e., Distinguished] to be untrustworthy" (Hambleton, et 
al., 1995, p. 6). 
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The OEA-sponsored panel made 12 recommendations to improve KIRIS and the 
accompanying accountability system. A number of these were suggestions that more research 
be done on the KIRIS system (e.g., validation studies) and better documentation be done of 
procedures (e.g., of how performance standards are set). Three recommendations were 
relatively specific: that portfolio data not be used in the accountability index, (b) that any 
assessment data used for accountability purposes be scored externally to the schools from 
which they are collected, and (c) that multiple-choice items should be used to enhance 
validity of the assessments. 

The Evaluation Center, Western Michigan University. The OEA-sponsored 
measurement panel largely focused on psychometric issues in their study. In terms of this 
review, their work is probably best categorized as dealing with the question of the 
implementation quality of KIRIS. Another major evaluation of KIRIS published in 1995 also 
is worthy of discussion in this regard— the evaluation of KIRIS performed by researchers from 
Western Michigan University and sponsored by the Kentucky Institute of Education Research 
(KIER) (The Evaluation Center, 1995). This evaluation had a somewhat different focus than 
the OEA study. Nevertheless, it has some relevance for assessing implementation quality. 

The Western Michigan evaluation was based on document analysis, extensive 
interviews with key informants about the KIRIS system, surveys of selected groups (e.g., 
assessment coordinators in school districts), and focus groups held in several parts of the 
state. Most work was pefformed between April and December 1994. 

The evaluators addressed six topics related to KIRIS and the accompanying 
accountability program: (a) consistency with legislative mandate, (b) understanding and 
confidence of stakeholders, (c) involvement of educators in design and implementation, (d) 
accuracy, accessibility, and clarity of documentation, (e) impact of accountability policies on 
students, teachers and schools, and (f) technical adequacy. The first topic, consistency with 
legislative mandate, was already discussed in the section of this review on basic 
implementation. In this section on implementation quality, all of the remaining topics will be 
discussed except item (e), on impact of KIRIS, which will be discussed later. 

Related to understanding and confidence of stakeholders, the evaluators found that the 
specifics of rewards and sanctions for schools were known only to a small number of people, 
mostly Department of Education personnel, superintendents, and people high in the 
bureaucratic hierarchy. (It should be noted that this conclusion was based on data collection 
in late 1994, months before any reward money was distributed.) Further, it was found that 
educators, parents, legislators, and the general public "have serious questions concerning the 
legitimacy, validity, reliability, fairness, and usefulness of the KIRIS assessment" (The 
Evaluation Center, 1995, p. 6). 

The evaluators found that teachers and principals were involved in the design and 
development of KIRIS. However, some teachers were unaware of such involvement and 
"perceived that questions on the assessment were constructed by outsiders with little or no 
knowledge of Kentucky" (The Evaluation Center, 1995, p. 6). 
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Regarding the accuracy, accessibility, and clarity of documentation of KIRIS, the 
evaluators identified several shortcomings. They stated that there will be a growing need for 
documentation. They also stated that there is considerable amount of technical information on 
KIRIS, but that it hard to access. Furthermore, related to what is now available, "the 
technical reports do not provide a complete perspective on the weaknesses as well as the 
strengths of the KIRIS assessment results and on the accountability index" (The Evaluation 
Center, 1995, p. 6). 

Related to the technical adequacy of KIRIS, the evaluators judged KIRIS assessment 
tasks to be well crafted and they found that open-ended questions met the standards for such 
items. Regarding the source of data by which schools are compared, the evaluators had a 
major disagreement with KDE. School accountability calculations with KIRIS data are 
performed using different cohorts. For example, a primary school is assessed by comparing 
results from this year’s fourth graders to those fourth graders who were in the school two or 
three years ago. This cohort (or cross-sectional) design means that accountability judgments 
are partly related to instructional practices in the school and partly the result of differences in 
cohorts who were measured (e.g., difference in socioeconomic status, academic ability, 
parental involvement, and a host of other variables). The Western Michigan evaluators 
concluded that a longitudinal evaluation approach would be better than a cohort approach. In 
the latter, the same students in a school would be tracked through the grades and their 
progress periodically compared. They stated that the longitudinal approach was favored by 
the great majority of district assessment coordinators and district superintendents. 

The evaluators expressed a belief that KIRIS should include multiple-choice items, and 
not depend solely on open-response items and other assessment formats. At the time the 
report was being written, it had been proposed that the 1994-95 KIRIS not have any multiple- 
choice items. In their view, measurement validity is enhanced by inclusion of the latter 
because students must then demonstrate knowledge and skills in a variety of ways. 

The Western Michigan evaluators stated that "the reliability of the accountability index 
is problematic for us" (The Evaluation Center, 1995, p. 9). This was based on data from the 
KDE in which statistical procedures were used that seemed to require unrealistic assumptions 
about data (e.g., that students were a "fixed" factor in the calculation of a generalizability 
coefficient). Because of the unreliability in two types of KIRIS assessments, portfolio scores 
and performance event scores, the accountability index is largely dependent on open-response 
item data. The evaluators stated that "we question whether the combination of these two 
components and the open-ended questions, which do evidence good reliability, give the 
Commonwealth a sufficiently reliable index for administering rewards and sanctions to 
schools" (The Evaluation Center, 1995, p.9). 

Similar to the OEA measurement evaluation panel, the Western Michigan evaluators 
issued a number of recommendations. There were 18, and most can be inferred from of the 
findings reported above (not all will be summarized). For example, the evaluators called for 
the reporting of more complete and detailed information on KIRIS. They also recommended 





better training of teachers, especially in portfolio scoring, but also in ways of incorporating 
KIRIS-like activities in regular instruction. 

Half of the recommendations related to technical issues. The evaluators recommended 
that more resources be devoted to performing technical studies on KIRIS, and that there be 
improved reporting of technical information. They recommended that reliability estimates of 
the accountability index be calculated using several different assumptions, so that it can be 
estimated how the assumptions affect the values obtained. Related to reliability calculations, it 
was recommended that KDE consult with a national expert in generalizability theory. 

Regarding types of assessment items,, the evaluators recommended that: (a) 
performance events continue to be included in the assessment, (b) multiple-choice items 
continue to be used, and (c) other types of performance items (e.g., oral presentation) be 
considered for inclusion. The evaluators called upon KDE to "provide evidence to 
demonstrate that the accountability index has a level of validity sufficient for use in high 
stakes decision-making ... if the necessary level of validity is not attained, do not continue 
to use the index . . . until it is improved" (The Evaluation Center, 1995, p. 1 1). Finally, it was 
suggested that the KDE consider using a longitudinal design rather than a cohort design to 
assess school change. 

Other research. Several other reported studies deal with quality of implementation of 
KIRIS. In size and scope, none of these approach the magnitude of evaluations performed by 
the OEA-sponsored panel and the Western Michigan group. Nevertheless, these other studies 
have value in illuminating various aspects of assessment. 

A paper by Cunningham (1995) provided a general desciiption of KIRIS and a critique 
of the first accountability cycle involving its use. Cunningham was critical of many aspects 
of KIRIS including its dependence on assessment formats like open-response, that are more 
expensive than multiple-choice items, but yield scores that are psychometrically similar. 

Cunningham criticized the KIRIS approach to calculating the reliability of school 
accountability scores. The formula for generalizability coefficients reported in ASME (1994a) 
assumed that error variance was completely estimated by the interaction of schools and testing 
tasks-an assumption that "seems quite indefensible" (Cunningham, 1995, p. 6). Furthermore, 
ASME assumed that student variability within schools constituted a fixed effect for purposes 
of estimating error variance. The latter assumption results in lower variability than if students 
were treated as a random factor, and, therefore leads to a higher generalizability (reliability) 
coefficient. 

Cunningham was critical of the separation of KIRIS from the state’s 57 academic 
expectations. In his view, this indicates the lack of content-related validity of KIRIS. 

Finally, he concluded that KIRIS has weak evidence of construct-related validity. This 
conclusion was based on correlational data from all the schools in the state. For example, the 
correlation between the accountability change score from 1992 to 1993 and accountability 
change score from 1993 to 1994 was -.3231 for elementary schools. Negative correlations 
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were also found for the middle grades and high school. Cunningham interpreted this to mean 
that "the improvement in scores is consistently inconsistent" (p. 12) and simply a reflection of 
changes in cohorts of students that are measured rather than real educational improvements. 

A series of three papers by Strong and Sexton (1995a, 1995b, 1995c) reported research 
examining the relationship between KIRIS scores and American College Testing (ACT) 
college placement test scores. Data from 1993 high school seniors were studied— specifically 
test scores in mathematics, reading and science. All regions of the state were sampled, with 
the 33 responding schools constituting a 20% return rate on the survey. 

Results reported for mathematics were representative of very similar results for the 
other two subject areas. The authors stated that "the Kentucky performance tests do not 
adequately discriminate differences in mathematics abilities, as measured by the ACT, at 
either the upper or lower ends of the rating scale (Strong & Sexton, 1995, p. 2). For 
example, 27% of students judged to be Novice in KIRIS mathematics (the lowest category) 
had ACT mathematics scores ranging from 1 8 to 36 on the ACT. The latter range represents 
moderate to very high levels of achievement. 

This same pattern occurred for reading and science. Hence the authors concluded that 
a student could do poorly on KIRIS mathematics, reading, or science but still do very well in 
the corresponding ACT scores in these areas. The authors took this as evidence of low 
validity of KIRIS. 

Strong and Sexton were not the only researchers to focus on ACT scores. The 
publishers of the ACT college entrance test themselves performed a study of recent Kentucky 
students (American College Testing Research Division, 1994). Data from this study are cited 
in KDE (1995), the OEA-sponsored evaluation (Hambleton, et al., 1995), and the KIER- 
sponsored Western Michigan evaluation (The Evaluation Center, 1995) However, the ACT 
study is worth examining on its own merits. 

In the first part of the report, an analysis was provided of how many ACT-tested 
students in Kentucky from 1991 to 1993 had taken recommended "core" courses (college 
preparatory sequences of high school courses in English, mathematics, science and social 
studies). The authors of the report issued a caution: "The trend that should be most 
disturbing to Kentucky, and warrants careful consideration, is that Kentucky is the only state 
in the nation whose ACT-tested graduates has shown a drop in the percent of graduates 
completing the ACT recommended core, compared to the other 49 states whose percent of 
graduates completing the ACT core sequence has increased steadily over the last five years" 
(American College Testing Research Division, 1994, p. 4). 

Regarding the correlation between the 1991-92 KIRIS tests and ACT tests, the authors 
of the report state that there exists a positive relationship between the measurements. 
However, other than the Distinguished level of student ability, the evidence for the other 
levels (i.e., Novice, Apprentice, Proficient) is said to be "somewhat suspect because of the 
nondiscriminating power of the Kentucky tests" (American College Testing Research 
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Division, 1994, p. 7). As a consequence, it was recommended that the Kentucky tests not be 
used for decision-making about individual students. 

In a study of the Kentucky accountability system, Hughes and Craig (1994) reported 
results of an investigation of 1 1 elementary schools from a district in western Kentucky. 
Teacher teams within the schools were rated by two persons who were working as program 
facilitators in a federally sponsored staff development project. Teacher teams were rated on 
the degree to which critical attributes of a multi-age, multi-ability primary program had been 
implemented. Attributes included the following: (a) an integrated curriculum, (b) 
opportunities for cooperative learning, and (c) use of whole language instruction. Data were 
averaged to yield a single implementation score for each school. 

In addition, information on KIRIS performance was also collected: fourth grade 
scores for 1991-1992 and 1992-1993 in reading, mathematics, science, social studies, and 
writing. Also reported were the percentage of student^categorized as proficient or 
distinguished in reading, mathematics, science, and social studies for the academic years 
1991-1992, 1992-1993, and 1993-1994. 

Hughes and Craig concluded that there was wide variability in the extent of 
implementation of critical attributes of the primary program. Furthermore the authors 
concluded that the fluctuations in both the implementation scores and KIRIS data meant that 
"it has not been possible to establish clear positive relationships between implementation and 
KIRIS results" (Hughes & Craig, 1994, p. 7). 

Under KERA, the school is used as the unit of analysis because schools differ in 
performance and the authors of the reform law believed that performance levels should be 
differentially rewarded. Hughes and Craig concluded that, compared to variance among 
schools, "just as much variability probably exists among teams of teachers within schools in 
implementing the multi-age, multi-ability primary program" (1994, p. 7). 

Summary. What can be concluded regarding the quality of implementation of KIRIS 
and its accompanying accountability systems? No single evaluative word can be applied to 
these systems— the quality is uneven. Examination of the research and evaluation data reveals 
areas of strength as well as areas of weakness. 

On the one hand, it is clear that extraordinary efforts have been made to construct a 
student assessment system that has no precedent or parallel in the U.S. Furthermore, the 
system has definitely had effects— many positive— on students and teachers. Effects of 
assessment will be discussed in later sections of this review. 

However, questions remain about key features of both the school accountability system 
and the KIRIS assessment system. To this reviewer, one of the most troubling relates to the 
use of an accountability index to reward schools, or to label them as needing assistance. 
Empirical research on the decision accuracy of the index has not been reassuring (see chapter 
3 of Hambleton, et al., 1995). The limited research that has been done on this question 
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suggests that a substantial number of schools were put in a low category of performance 
because of essentially random factors unconnected with instructional efforts of teachers in 
those schools. For example, the OEA panel cited one analysis which suggested that one out of 
three schools at grade 4 were erroneously classified "in crisis." 

It must be acknowledged that technical data cannot answer every question about the 
adequacy of KIRIS and the accountability system. This reviewer believes that a 33% error 
rate for accountability decisions is unacceptably high. However, what one person considers 
unacceptable might not be considered so by other observers of the system. Regarding how 
many schools are erroneously classified, Hambleton, et al. (1995) note that "A policy 
judgment is required to determine whether the estimated error rates are acceptably low" (p. 3- 
3). 



Another problem area, one mentioned by the OEA panel, the Western Michigan 
evaluators, and Cunningham (1995), concerns the method used by ASME and KDE to 
calculate reliability estimates of KIRIS scores. Most such estimates only took into account 
sampling of students, not sampling of items and other factors. As a result, reliability 
estimates were higher than they should have been, because sources of measurement error were 
ignored. This implies that data presented by KDE would not have been as positive had more 
comprehensive reliability analyses been performed. 

Complete reliability analyses— reflecting as many sources of error as can be feasibly 
estimated— will eventually be needed if KIRIS data are to be defensibly used to make 
decisions. For example, Hambleton, et al. (1995) stated that "School-level assessment results 
are reported by subject and grade. If those results are to be used to make inferences about 
improvements in school performance in a subject area from one year to the next, estimates of 
measurement errors associated with those scores should take into account cohort, task, and 
rater differences" (p. 3-2 /). 

KDE (1995) made a start toward more realistic estimates of reliability. Earlier in this 
review, it was noted that chapter 13 of KDE (1995) contains generalizability coefficients 
reflecting sources of error in addition to students. However, analyses will need to be even 
more detailed (e.g., at the level of each school subject) in order to provide schools with fully 
informative data about student performance. 

Regarding the types of assessment formats used in KIRIS, the most difficult questions 
relate to multiple-choice items and portfolios. Both of the large-scale evaluations of KIRIS, 
by the OEA measurement panel and the Western Michigan evaluators, recommended that 
multiple-choice items continue to be used in KIRIS as a way of enhancing reliability and 
validity. The comments of the OEA panel regarding multiple-choice items are especially 
significant, because the panel was charged with examining exclusively psychometric issues. 

The panel’s recommendations on multiple-choice items were based on what they 
viewed as the improvements that would result to both accountability decisions and the 
reliability of individual test scores. Regarding school accountability decisions, Hambleton et 
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al. (1995) stated, "Misclassification errors could be reduced by administering a larger number 
of test questions to each student. Misclassification errors could also be reduced by including 
the multiple choice items in the calculation of accountability indices" (p. 3-2). Regarding 
effects on individuals, Hambleton et al. (1995) stated, "Reliability of both the student and 
school subject area scores could be improved by using both multiple-choice items and open- 
response tasks to obtain scores" (p. 3-4). * 

Portfolios in writing and mathematics have been used with thousands of Kentucky 
students, but questions remain about the technical adequacy of portfolio scores. The Western 
Michigan evaluators recommended that portfolios continue to be used, that they be scored by 
local teachers, and that efforts be made to upgrade teacher training in portfolio scoring. The 
OEA measurement panel recommended that portfolios be used for instructional enhancement 
only (i.e., that scores not be included in school accountability index) and that they be scored 
by outside readers rather than local teachers. In effect, only the OEA panel believed that there 
should be changes in the way portfolios are handled. 

Portfolios constitute the assessment format in which there is the sharpest contrast 
between emerging criteria of assessment quality (e.g., consequential validity) and traditional 
psychometric criteria (e.g., inter-rater reliability). What a person values has a lot to do with 
how that person perceives portfolios. Using traditional criteria, portfolios have weaknesses. 

But it is clear they can have positive instructional effects. Perhaps it is not surprising that the 
OEA panel and the Western Michigan evaluators arrived at different recommendations about 
portfolios. The OEA panel focused almost exclusively on traditional forms of reliability and 
validity, while the Western Michigan group gave considerable weight in its evaluation to 
reactions of teachers and students. 



What have been the effects of the program on stakeholders? 

The question of effects pertains to the broad question of what has happened to people 
as a result of KIRIS. For example, how much have teachers changed their behavior as a 
result of KIRIS and accountability? Several studies or parts of studies bear on this question. 

Advanced Systems for Measurement and Evaluation and Kentucky Department of 
Education. Chapter 15 of the comprehensive report summarizing research by the state’s 
testing contractor and Kentucky Department of Education (KDE, 1995) deals with 
consequential validity. The latter term is attributable to the measurement theorist Samuel 
Messick, who argued that the interpretation of what a test means (its consu act validity) is 
separate from the use of test information (consequential validity), and that both are important 
in judging a testing system. According to the authors of KDE (1995), the intended goals of 
KIRIS were to provide: (a) goals, standards, and criteria for educational achievement, (b) 
useful information on the progress of schools toward meeting goals, and (c) information on 
potential biases and differential impact of the assessment. The authors note that relatively 
little research has been done on classroom practices and teacher development in Kentucky, 
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and that the results that have been reported are hard to attribute to one particular aspect of the 
reform law. 

Regarding the consequences of KIRIS on instructional practice, the authors of KDE 
(1995) stated that "over 140,000 students in each of grades 4, 8, and 12 produced writing 
portfolios and mathematics portfolios" (KDE, 1995, p. 216). These required that students 
work with teachers on instructionally-relevant tasks. Other evidence cited included the fact 
that over 85% of schools photocopy student work from accountability tests and that 65% of 
schools in the state purchase "continuous assessments" (the latter are KIRIS-like tests that can 
be used for practice in the non-accountability grades). Also cited was a study on the use of 
portfolios, performed by Appalachia Educational Laboratory (AEL, 1994), which will be 
discussed later in this review. 

KDE (1995) mentioned the impact on professional development as one consequence of 
KIRIS. Examples have been 300 educators in an "assessment fellows" program, 100 
"distinguished educators" trained to help low achieving schools, and 300 teachers participating 
in summer rescoring of portfolios. Also mentioned were thousands of teachers viewing 
professional development telecasts on statewide educational television and participating in 
traditional workshop activities. 

KDE (1995) cited a report evaluating technical assistance for Chapter I programs 
(Policy Studies Associates, 1994) which named Kentucky as the only case among nine sites 
that were studied where technical assistance centers were working with local educators to 
develop new forms of assessment. Also cited was another evaluation involving Kentucky: a 
study of how teachers in different parts of the country are using mathematics portfolios 
(McCollum, 1994). In the latter, teachers expressed both positive and negative feelings, but 
had enthusiasm about the instructional potential of portfolios. 

How much impact has KIRIS had on support for educational reform in the state? The 
authors of KDE (1995) are uncertain: "It is not clear whether the KIRIS accountability 
system is having a positive or detrimental effect on support for educational reform in general" 
(p. 219). In the view of the authors, KIRIS embodies certain values and beliefs that were 
held oy the authors of the reform law— beliefs about learning and human ability that not 
everyone shares. Thus, opposition to KIRIS and KERA may simply be the inevitable result 
of a conflict in values. 

In a final section on the impact of KIRIS on general educational reform, the authors of 
KDE (1995) cite a study of state assessment systems sponsored by the National Science 
Foundation (NSF) (Policy Studies Associates, 1995). It was reported that Kentucky was one 
of the few states that met three criteria indicating that a state’s assessment system could 
contribute to systemic educational reform in mathematics and science. 

The authors of KDE (1995) stated that a goal of KIRIS was to provide educators with 
useful information on the progress of schools toward meeting goals. This has been 
accomplished by reports of a school’s accountability index, and comparison with its target for 
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performance (i.e., threshold). In addition, individual student reports have been sent to districts 
for distribution. Although schools and teachers have reported "using the score reports in a 
variety of ways consistent with the intent of KIRIS" (KDE, 1995, p. 222), there have been 
requests for additional kinds of information. KDE (1995) cited the previously reviewed 
KIER-sponsored evaluation (The Evaluation Center, 1995) as one source of these suggestions. 
The Western Michigan study mentioned information needs that are currently not addressed by 
KIRIS, including: (a) national comparative data on Kentucky students, and (b) data for 
schools that can be used for instructional program evaluation. The authors of KDE (1995) 
concluded that providing such types of information "would require substantial changes in 
design and operation of KIRIS" (pp. 222-223). 

% 

The final section of Chapter 15 of KDE (1995) deals with the issue of fairness. 
Statistical analyses were performed examining the influence of several factors on a school’s 
accountability score— including geographic region, racial/ethnic composition, student economic 
status, initial baseline score, school size, and grade level organization. Based on their 
analyses, the authors concluded that "the Kentucky assessment program appears to be fair in 
that rewards and assistance were distributed across these dimensions without statistically 
significant unevenness" (KDE, 1995, p. 223). The exception was grade level, because more 
elementary schools received rewards than middle schools or high schools. However, this 
differential was attributed to real educational differences among the grades rather than bias. 

A correlational/prediction study led the authors to their conclusions. The 1,247 
schools in the state were measured on 10 variables, including school result (ranging from 
Reward School, coded 5, to School in Decline, coded 1), and five different accountability 
measures (e.g., the baseline score from 1991-92, the average 1993 and 1994 index). Also 
included were measures of ethnicity, wealth, number of students enrolled in accountability 
grade, and accountability grade (e.g. 4, 8, 12). 

A multiple regression analysis was performed with Percent of Improvement Goal 
(PCTGOAL) as the dependent variable. This was a relative measure of how much the school 
had improved (PCTGOAL = (Overall Index - Baseline)/(Improvement Goal - Baseline)). Five 
predictors were used in the equation: Accountability Grade, 1991-92 Baseline Score, 
Percentage of Minority Students, Percentage of Students Receiving Free and Reduced Lunch, 
and Size of School. The squared multiple correlation coefficient (i.e., R square) was .17 with 
the five predictors, meaning that they collectively accounted for 17% of the variance in the 
dependent variable. The authors took this to be evidence of fairness, because Percent of 
Improvement Goal was not much affected by the predictors. If a large amount of variance 
had been detected, it would have been evidence that certain systematic factors (unrelated to 
instructional efforts being made in the school) were determining how much improvement 
occurred. 

When the Percent of Improvement Goal was regressed on region of the state (eight 
regions of the state dummy-coded as predictors), the resulting R square was .01, indicating 
only 1% of the variance was accounted for. Again, this was interpreted as lack of bias in 
accountability outcomes regarding regions of the state. 
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The strongest relationships between any predictor and a school outcome variable 
involved school grade. For example, the variable ACCGRADE (accountability grade) and 
School Result correlated -.31. This meant that the lower the grade, the higher (better) the 
school outcome. In other words, elementary schools did better than middle schools and high 
schools. The authors of KDE (1995) list several possible reasons why this occurred. 

Theories include: (a) greater impact of KERA on young children than on older students, and 
(b) personality differences and professional development differences among teachers at 
different educational levels. 

Separate regression analyses, similar to that explained above, were calculated for 
schools at different accountability grades (e.g., 4, 8, 12). Results were not very different than 
previously stated. Factors such as initial baseline score, wealth, or enrollment size did not 
have much relationship to Percent of Improvement Goal. 

The authors of KDE (1995) examined the simple r values (univariate correlations) 
between Baseline Score and Percent of Improvement Goal. For all schools in the state, the r 
was -.20, but results varied by grade. The r values were: Grade 4, r = -.08; Grade 8, r = .13; 
Grade 12, r = -.23. A negative correlation means the lower the Baseline Score, the higher the 
Percent of Improvement Goal. The authors concluded that "schools with initially lower 
scores may have a slightly easier time making the progress needed to achieving reward status" 
(KDE, 1995, p. 228). 

A similar analysis was pursued with the variable Student Economic Status (measured 
as percent of children on free/reduced lunch). For all schools in the state, the r between 
Student Economic Status and Percent Improvement Goal was .06, with results varying by 
grade. The r values were: Grade 4, r = -.06; Grade 8, r = -.26; Grade 12, r = -.11. A 
negative correlation means the lower the number of children on free lunch, the higher the 
Percent of Improvement Goal. The authors believed that the relationships were not strong. 

For example, they characterized the largest r value as follows: " a correlation of -.26 at grade 
8 means that percent free lunch only accounts for about seven percent of the variance in 
Percent of Improvement Goal" (KDE, 1995, p. 228). 

The authors of KDE (1995) present additional information buttressing their argument 
that wealth played little role in which schools got rewarded. They present tables listing every 
school district in the state, ranked from highest to lowest in Percent of Improvement Goal (a 
district-wide average of the same variable used in the regression analyses just reported). The 
districts were then ranked by taxable property value per student— a measure of district wealth. 
In a' oting the newspaper that did the rankings (the Lexington Herald-Leader) the authors of 
KDE (1995) noted that 1 1 of the 20 highest scoring districts and half of those getting rewards 
ranked in the bottom half in district wealth. This meant that poorer districts were not 
disadvantaged in getting rewards for achievement gains. 

The final page of the chapter addresses the issue of how much enrollment of minority 
children affected school outcomes. For all schools in the state, the r between Racial/Ethnic 
Minority Enrollment and Percent Improvement Goal was -.13, with results varying by grade. 
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The r values were: Grade 4, r = -.22; Grade 8, r = -.08; Grade 12, r = .03. A negative 
correlation means the lower the percentage of minority children, the higher the Percent of 
Improvement Goal. The authors did not believe the results were a cause of concern since the 
largest correlation, -.22 in grade 4, was "small in magnitude" (KDE, 1995, p. 235). 

In summary, chapter 15 of the KDE (1995) report contains a variety of evidence on 
the consequential validity of KIRIS and the accountability system. The picture that is 
presented suggests impact on teachers, primarily in terms of professional development. 
Regarding students, schools, and teachers, it was argued, based on regression analyses, that 
background variables (including student wealth, region of the state, and student ethnicity) 
were not strongly connected to school reward status. 

KIER Performance Assessment Configuration Component (PACC) map study. In 
terms of the effect of KIRIS on teachers, one of the more useful studies on the topic was 
published in late 1995. This was a statewide study of the implementation of performance 
assessment in classrooms (Matthews, 1995). It was one of a series of six studies examining 
the implementation of different aspects of school reform sponsored by the Kentucky Institute 
for Education Research (KIER). 

The study is being reviewed in this section, because it was largely an examination of 
how teachers reported behaving in the classroom, and how they reported instruction had 
changed. Although it was labeled an implementation study, it could as easily have been 
labeled a study of the consequential validity of KIRIS. 

The primary purposes of the study by Matthews (1995) were: (a) to determine the 
extent of implementation of KIRIS, and (b) to develop recommendations for improving its 
implementation. Eight regions of the state (corresponding to each of the Regional Service 
Centers) were used as strata for sampling purposes. Within each region, four schools were 
randomly sampled: two primary schools, one middle school, and one high school. Six 
teachers in each school were randomly sampled to participate in a structured interview and to 
complete a questionnaire (192 teachers statewide). In addition, the other teachers in each 
school (total of 308) completed only the questionnaire. 

The 192 teachers who were most intensively studied were interviewed by trained data 
collectors, most of whom were professors from state universities. The interviewers used a 
Performance Assessment Configuration Component (PACC) map as a guide. Using this 
document, interviewers asked teachers about various classroom practices related to 
performance assessment. For example, one set of items required teachers to report how often 
they used different types of assessments (e.g., written open-ended, portfolio tasks or prompts). 
There were eight major components of performance assessment that were studied. Interview 
questions were designed for each component, and, in addition, interviewers gave ratings to 
each component, based on judgements of teacher responses. 

The study revealed that teachers varied considerably in their understanding and uses of 
performance assessment; there was considerable variation, even among teachers within the 
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same school. It was concluded that "more than 60 percent of the teachers interviewed 
appeared to be using performance assessment on a regular basis" (Matthews, 1995, p. 7), with 
open-ended tasks and portfolio tasks the most commonly used. Of the teachers using 
strategies, most were in the accountability grades rather than non-accountability grades. 
However, many teachers "separate the preparation for KIRIS testing from the assessment 
strategies that are normally used in the classroom" (Matthews, 1995, p. 7). 

Two parts of the results are especially relevant to the question of the impact of KIRIS. 
First, based on the PACC map data, 65% of teachers were judged to be using performance 
assessment tasks, most or all of which, had a clear link to the state’s 57 academic 
expectations. Second, 53% of teachers stated, that they "largely or always" used assessment to 
drive subsequent instruction and instruction to drive subsequent assessment (Matthews, 1995, 
p. 14). 



Recommendations of Matthews (1995) included that more resource material be made 
available to teachers to help provide examples of assessment and that increased professional 
development be provided on how to design challenging and meaningful assessment tasks that 
can be used to organize and drive instruction. 

Overall, Matthews (1995) presented a complex picture of the instructional effects of 
KIRIS and accountability. There are some successes, but there are also areas where progress 
could be made. Probably the area where this is most true relates to developing policies and 
procedures that help teachers meaningfully integrate performance assessment into the process 
of instruction. 

The Evaluation Center, Western Michigan University. Of the two major evaluations of 
KIRIS published in 1995, only the KIER-sponsored study (The Evaluation Center, 1995) 
contains a section on the impact of KIRIS. The authors stated that teachers, district 
assessment coordinators and superintendents all reported that student writing had improved in 
the state. The authors concluded that portfolios "have great instructional potential" (The 
Evaluation Center, 1995, p.7), but concluded that they are less reliable than other assessment 
formats. The authors also concluded that the time spent by teachers in receiving training and 
then scoring portfolios is reasonable and is probably beneficial as inservice. 

The KIRIS-based accountability index does not reflect certain input factors that are 
outside the school’s control (e.g., student mobility). However, a balancing factor in the use 
of the formula is the fact that schools can appeal their accountability outcome. One problem 
with the accountability index is its focus on the school as the unit of education. It is not 
geared to timely feedback or instructional improvement at the classroom level. 

The Western Michigan evaluators stated that "there is disagreement whether the 
system of rewards and sanctions will improve the quality of education in the state of 
Kentucky" (The Evaluation Center, 1995, p. 7). While school district assessment coordinators 
believe improvement will occur, superintendents are less sure, and teachers believe they will 
not cause improvement. 
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Other studies. A dissertation by a University of Louisville student addressed 
instructional impact of KIRIS. Clifford (1995) surveyed over 400 teachers in Jefferson 
County, Kentucky in June 1994. Jefferson County includes the city of Louisville and its 
immediate suburbs. Jefferson County Public Schools (JCPS) is the largest school district in 
the state. 

The purpose of the study was to determine whether teachers would report changes in 
instructional practices in the last three years (i.e., since KIRIS assessments have been used). 
Approximately 73% of respondents reported either an increase or a large increase in using 
problem-solving activities as an instructional. strategy. About 76% of respondents reported 
either an increase or a. large increase in using cooperative leaming/group v/ork as a teaching 
technique. Clifford (1995) concluded that "Large percentages of teachers reported an increase 
in the use of open-ended questioning, writing assignments, problem-solving activities, and 
cooperative learning .... Elementary teachers, female teachers, and teachers in the assessed 
grades reported making significantly more progress than others" (p. iv). In general, the study 
showed that teachers are reporting changes in instructional practices since KERA. 

Another recent dissertation directly dealing with Kentucky’s assessment system was 
completed by a University of Kentucky student (Vitali, 1993) who sought to determine the 
impact of performance assessment on teacher instructional practices. Some of the study’s 
major points were summarized by Guskey (1994b). 

Vitali collected his data during the 1992-93 academic year using interviews and 
questionnaires of teachers and classroom observations. Vitali found that "the performance- 
based assessment program . . . resulted in only modest changes in teacher instructional 
practices" (Guskey, 1994b, p. 53). Some teachers that were studied added some writing 
activities and made small changes in instruction, but most did nothing. Guskey contends that 
this was not due to lack of awareness of the new assessment system, but that it reflected 
teacher beliefs that they did know how to teach to the new assessments and that at this early 
point in the reform effort they had not had either the time or the opportunity to receive 
training in new instructional approaches. Guskey concluded that performance assessment 
alone is insufficient to change instructional practices but that new assessment systems must be 
accompanied by high-quality professional development opportunities. 

Summary. The research on the impact of KIRIS and accountability has been limited. 
Research has focused almost exclusively on teachers and other school personnel. No research 
has been done on the effects of KIRIS on parents and students (although, as will be seen in 
the next section of this review, survey research has been done on these latter stakeholders). 

The data on consequential validity provided by KDE (1995) constitute a mixture of 
purely information material (e.g., how many teachers attended workshops on assessment) and 
statistical analyses of accountability data. For example, regression analyses revealed weak 
associations between accountability outcomes and systematic factors like economic level of 
students and region of the state in which a school was located. Such analyses would appear 
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to be good news. However, questions have been raised about the reliability of both KIRIS 
tests and the accountability decisions that they support. Thus, any analyses using 
accountability indices as dependent variables must be viewed as tentative, and subject to 
changes in interpretation if reliability estimates decrease in the future. 

A number of investigators have collected evidence that teachers have altered their 
instructional practices. For example, the two largest studies involving impact on teachers, by 
Matthews (1995) and Clifford (1995), both supported the conclusion that many teachers are 
engaging in testing and instructional practices that are consistent with the spirit of the reform 
law. Reported practices included using non-traditional assessment formats and hands-on 
assessment materials. 

However, there is also evidence that the impact varies. Matthews (1995) reported 
considerable variability in the implementation of performance assessment from teacher to 
teacher. Clifford (1995) reported gender differences and grade level differences. If a major 
goal of KIRIS and accountability is to change instruction in every classroom of the state, then 
additional work needs to be done to accomplish this goal. The recommendations of Matthews 
(1995) are probably the most pertinent on this point. Professional development would appear 
to be the greatest need. Especially crucial is the need to help teachers better integrate 
instruction and assessment. 



What are the attitudes and perceptions of stakeholders toward program? 

The most comprehensive efforts to measure attitudes about KIRIS have been reported 
in several studies funded by the Kentucky Institute for Education Research (KIER). Large 
scale telephone surveys were completed in July 1995 (Wilkerson & Associates, Ltd., 1995) 
and July 1994 (Tom Wilkerson & Associates, Ltd., 1994). In addition, mail surveys were 
reported on school counselors (Craig, 1994) and school superintendents (Kentucky Institute 
for Education Research, 1994). All of these studies involved attitudinal questions dealing with 
the major aspects of KERA, with numerous items dealing with KIRIS and school 
accountability. 

The 1995 telephone survey (Wilkerson & Associates, Ltd., 1995) involved 859 
members of the general public, 451 public school parents, 100 school council parents, 609 
teachers, and 214 principals. Respondents were randomly selected to represent the state 
regarding geographic region and demographic variables (e.g., ethnicity). Similar sample sizes 
were used in 1994. 

When the 1995 samples were asked the question, "Is KIRIS working?" school 
personnel and school board members were less positive than school parents or members of 
the general public. For example, 32% of principals and 29% of teachers s. A it was "working 
well." However, 49% of school parents and 48% of the general public said it was working 
well. 



When the same questions were asked in 1994, there was a similar contrast between 
school personnel and non-school persons. However, support for KIRIS among school 
personnel had been higher in 1994 than 1995, e.g., 41% of principals and 37% of teachers in 
1994 said KIRIS was working well (Wilkerson & Associates, Ltd., 1994). 

In the 1995 sample, the contrast between various stakeholder groups was also evident 
on questions dealing with accountability. For example, respondents were asked whether they 
thought, "Both rewards and sanctions are essential to hold teachers and schools accountable 
for school learning." Only 22% of principals and 20% of teachers agreed with this statement, 
in contrast to 75% of school parents and 68% of the general public (Wilkerson & Associates, 
Ltd., 1995). 

From the standpoint of the future of KIRIS and accountability, the 1995 survey 
contained questions on a number of things that might be changed in the state’s systems. 

Several of the categories of respondents were asked to respond to the statement: "The 
portfolio program should be continued, and scores reported, but not for rewards and 
sanctions." Strong majorities of school-level educators agreed with this— 70% of principals 
and 72% of teachers. Even stronger support, 83% of principals and 85% of teachers, favored 
administering a variety of KIRIS assessments, including multiple-choice items, to determine 
which should be used in the future. Finally, very large majorities of respondents favored the 
state administering a nationally normed standardized test, not for accountability, but to 
determine the achievement levels of students. Over 85% of principals, teachers, and school 
council parents favored this. 

The KIER-sponsored surveys provide evidence that support for testing was not as 
great in 1995 as it was in 1994. Furthermore, KIRIS and accountability (rewards and 
sanctions based on student performance) are viewed much more positively by non-educators 
than by educators. When asked about how KIRIS could be changed, there were high levels of 
agreement on a number of possible changes. 

In another attitude study, Horizon Research International (1994) reported a telephone 
survey of 102 members of the Kentucky state legislature. In this study sponsored by the 
Office of Education Accountability (OEA), legislators were called in July 1994 and asked 
their opinion of KERA, especially the KIRIS system. In rating various aspects of the 
educational reform law, 70% of legislators believed that the assessment and accountability 
system have been "successful" or "somewhat successful." In rating the various assessment 
formats, the legislators perceived the accuracy of information to be highest for performance 
events, somewhat lower for writing and mathematics portfolios, and lowest for paper-and- 
pencil tests. However, this survey was completed before the release of evaluation studies on 
KIRIS, and it is likely that attitudes would be different if legislators were surveyed in 1995. 

In 1994, the Appalachia Educational Laboratory (AEL) reported two piece of research 
related to Kentucky’s assessment. In one AEL study, student attitudes toward KIRIS were 
obtained (Gregg, 1994). Students came from three "geographically different" school districts. 
From each district, a middle/junior high school and a high school participated in the study, for 
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a total of six schools. In each school, 10 students were randomly selected to be in a focus 
group and six students were randomly selected for individual interviews. 

Students reported that their education has changed since the passage of KERA: "The 
use of portfolios in math and language arts, an increase in writing and group work, and 
changes in the state testing system were the most frequently noted responses. Students 
believed that the greater emphasis on student writing occurred because of writing portfolios. 
Students were positive about these changes, though some worried that they might not be 
adequately prepared for college and their future careers" (Gregg, 1994, p.l). Students were 
aware that the various assessment formats in KIRIS placed an emphasis on the ability of test 
takers to express themselves. In addition, sojne students believed the KIRIS assessments 
contained difficult questions. 

For several years, a qualitative study was done of the process of school reform in 
four rural Kentucky school districts sponsored by the Appalachia Educational Laboratory 
(AEL). An early report of this was described by Coe and Kannapel (1991). However, most 
pertinent to assessment was a study published in 1994. As a follow-up to ongoing research on 
the KERA primary program, research was reported by the Appalachia Educational Laboratory 
(AEL) that dealt with the relationship between assessment and instruction in both accountable 
grades (4 and 8) and a nonaccountable grade (grade 5) (Appalachia Educational Laboratory, 
1994). Researchers interviewed 64 educators in 13 schools in four rural Kentucky districts: 

13 principals, 37 teachers, and 14 eighth grade students. Researchers also observed 73 
teachers working with students in the fourth, fifth, and eighth grades (with some combined 
classes including sixth grade students). 

AEL researchers found that KIRIS appeared to be the driving force behind 
instructional changes that were observed. The major instructional change was an increased 
emphasis on writing and the writing process (at both accountable and nonaccountable grade 
levels). In general, those teachers who received the most training on writing and portfolios 
were the most enthusiastic about the emphasis on writing, but teachers gave a mixture of both 
negative and positive comments about this emphasis. The most common negative teacher 
comment was "portfolios are time-consuming and burdensome" and the most common 
positive comment was "students’ writing and thinking skills have improved tremendously" 
(Appalachia Educational Laboratory, 1994, p. 1). In addition to the instructional emphasis on 
writing, researchers noted the existence- of the following instructional strategies: "traditional 
and nontraditional uses of textbooks and worksheets, group work, hands-on activities, and use 
of authentic literature to teach reading" (Appalachia Educational Laboratory, 1994, p. 2). The 
AEL investigators saw little evidence of school-wide instructional planning— only one school 
out of 13 seemed to have such a plan. AEL researchers stated their opinion that "teachers in 
the upper-elementary grades need ongoing and focused professional development on 
instructional strategies that will help students achieve KERA goals, and on how to incorporate 
assessment techniques into regular classroom instruction" (Appalachia Educational Laboratory, 
1994, p. 2). 
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Another study in which attitudes toward assessment were examined was a qualitative 
study of teachers, parents and administrators. The study was conducted in nine school 
districts serving urban, suburban, and rural communities around Louisville (Whitford, 1993). 
Teachers were interviewed about KERA in general, including their reactions to performance 
assessment and accountability. Generally speaking, mixed reactions were reported. The 
overall impression given by these interviews was tjaat these teachers generally supported the 
concept of performance assessment, if not the specific ways the concept was being 
implemented. 1 

Summary. Attitudes toward KIRIS and the accountability system were found to be 
highly affected by the role of the respondent,(e.g., teacher, principal, parent). This should be 
expected, given the fact that school personnel are the direct recipients of rewards or sanctions. 
Thus., it is not surprising that educators expressed less enthusiasm for accountability than 
other groups, like school parents or the general public. (Perhaps fear of sanctions, or 
embarrassment at low school performance cause attitudes to be negative, and these negative 
attitudes are relatively unaffected by the potential receipt of monetary rewards.) 

For several types of stakeholders, especially educators, attitudes toward assessment and 
accountability have become more negative over the past year. There are two likely 
contributing factors to this: (a) the experience of many educators of working in a school that 
received no cash rewards, and (b) the negative publicity about KIRIS and accountability 
following the release of the Western Michigan study and the OEA measurement panel 
evaluation. 

An interesting Finding about 1995 attitudes was that strong agreement was found 
among several groups for changes in assessment. This included strong support for adding a 
nationally normed test to the state’s array of assessment instruments. 



What other questions have been addressed? 

A number of sources will now be cited that in some way bear on KERA assessment, 
but have as their main focus something other than empirical research. These include papers, 
articles, and books that focus on issues like the following: (a) historical background of 
performance assessment in Kentucky, (b) descriptions or critiques of performance assessment, 
and c) policy issues related to performance assessment. 

Important documents that provide a background for KIRIS and KERA-mandated 
assessments are papers, studies, and reports produced by the Kentucky Council on School 
Performance Standards. This group started its work even before KERA was passed into law, 
but many of its ideas have been embodied in provisions of the law. Especially significant is 
the report of the Assessment Task Force (Council on School Performance Standards, 1989) 
that provided a rationale for Kentucky’s shift away from traditional tests and toward newer 
assessment formats. 
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Almost immediately after KERA passed, the University of Kentucky, the University of 
Louisville, Eastern Kentucky University, and Western Kentucky University collaborated on a 
state-wide conference in Lexington that dealt with Kentucky’s high-stakes performance 
assessment system. The proceedings of the conference (Mason & Kifer, 1991) contain 18 
chapters on the pitfalls and promise of performance assessment, many of which continue to 
be relevant today. In a similar vein, a 1992 issue of the periodical, New Directions for 
Education Reform, a publication of Western Kentucky University, contained eight articles on 
performance assessment. These, dealt with such matters as definitions of performance 
assessment and problems of implementing it in the classroom (Aschbacher, 1992; Galluzzo, 
1992; Kennedy, 1992; Kifer & Mason, 1992; Petrosko, 1992; Redfield, 1992; Winograd & 
Jones, 1992; Worthen & Leopold, 1992). 

The publishers of the newsletter called The Link devoted a special issue to alternative 
assessment (Appalachia Educational Laboratory, 1993). The issue included a summary of the 
use of alternative assessment in states of the Appalachian region, and citations to articles and 
reports about new forms of assessment, like portfolios. 

Cunningham (1993) wrote a paper critical of both the KIRIS assessment and the 
school accountability system. In Cunningham’s view, KIRIS tests measure primarily 
intellectual ability, not achievement. As such, they are similar to (but more expensive than) 
the Comprehensive Tests of Basic Skills (CTBS) and other multiple-choice standardized tests. 
Because he believes KIRIS tests measure intelligence rather than classroom knowledge, 
Cunningham contends that they are unfair to teachers. The assessment and accountability 
systems require teachers to change something not under their control— student intellectual 
ability. 



Descriptions of KIRIS assessment have appeared in many of the Kentucky’s 
newspapers. Articles have appeared to inform citizens of the nature of the new assessments 
and how they will be used in calculating an accountability index. A useful series of articles 
attempting to explain how the new assessments are different than traditional tests appeared in 
the Louisville Courier-Journal (Holland, 1992a, 1992b, 1992c). Newspapers have also 
reported test score results. For example, in fall 1993, KIRIS test score results for all schools 
in the state were published in a special section of the Louisville Courier-Journal ("KERA: 
Knowing the score," 1993). 

The annual reports of the Office of Education Accountability (OEA) have chapters 
dealing with assessment (Office of Education Accountability, 1992, 1993, 1994). Although 
limited in scope, they provide a perspective on assessment from the official state agency 
charged with monitoring the implement of the Kentucky Education Reform Act. 

A short book published in 1994 provides a useful general overview of KIRIS and the 
Kentucky accountability system (Guskey, 1994a). The book has chapters by individuals who 
have had a connection with the assessment reforms mandated by KERA. The authors provide 
the reader with a rationale, history, and description of the Kentucky assessment system, 
including comments on its strengths and weaknesses. 
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In two early chapters, Edward Kifer provides a history of the system and also includes 
a description of the components of KIRIS, and Peter Winograd and Karen Schuster Webb 
deal with the impact of assessment on curriculum and instruction reform. 

A chapter dealing exclusively with the accountability system was written by Scott 
Trimble, a Kentucky Department of Education official responsible for implementing some 
aspects of accountability. The chapter includes a clear description of how the various test 
formats contribute to the accountability index, and how the index translates into various 
decisions about school personnel~i.e., rewards or sanctions. 

Perhaps appropriately, the chapter is followed by a school district perspective on 
testing and accountability, provided by Ben Oldham, who was director of research for Fayette 
County (Lexington, KY) schools at the time the book was written. Oldham provides a 
candid exploration of the practical problems facing a local district that must implement the 
assessment system and live with the consequences of accountability. Oldham is cautiously 
optimistic about the potential of the system to improve student achievement: "compared to 
testing programs in the past, it seems to be a step in the right direction" (Guskey, 1994a, p. 

93). However, he points out several limitations to the system. These include a heavy 
emphasis on decision-making about schools rather than individual students, and the lack of 
national norms for the limited student reports that KIRIS provides. 

The final chapter of the Guskey (1994a) book is by Edward Haertel, a nationally 
known expert on assessment. Haertel acknowledged some distinctive strengths of KIRIS. For 
example, the assessment is part of a total school reform law which included greatly expanded 
funding for education. Assessment was not instituted as a stand-alone effort at school reform. 
Furthermore, it was designed with a phased implementation and a realistic 20-year time line 
for achievement of its ultimate goals. Other strengths of KIRIS include the inclusiveness of 
students who are assessed. For instance, disabled students are measured along with the non- 
disabled. Finally, Haertel praised the quality and comprehensiveness of the assessment tasks 
themselves— which provide students with meaningful activities and invite teachers to revise 
their curricula in creative ways. 

Among psychometric challenges identified by Haertel are several related to reliability 
of assessment, standard setting, and construction of the accountability formula. Haertel 
advocates resisting the pressure to interpret data at the level of individual students until such 
time as adequate reliability for individual decisions is established. Regarding standards of 
student performance (i.e., the categories novice, apprentice, proficient, and distinguished) the 
challenge is to create tests of equivalent difficulty as the years go by. Trying to develop new 
test items that equate in difficulty with previously used items is complicated by the fact that 
students are receiving instruction relevant to all of the test items. In addition, the meaning of 
standards vary from one subject area to another. For example, what is classified proficient in 
writing is different than what is classified proficient in mathematics. Finally, it is debatable 
how test scores should be weighted in arriving at a cognitive index which combines with 
noncognitive outcomes to form an accountability index. 
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Overall, the Guskey (1994a) book provides a useful overview of the KIRIS system. It 
should be enlightening both to Kentucky readers and to those from out of state who wish to 
understand the country’s first high-stakes school accountability system substantially based on 
alternative assessment methods. The book contains a combination of objective information 
about the system along with a realistic discussion of both its advantages and drawbacks. 

Another source that provides a general overview of KIRIS and accountability is a 
paper produced for a conference sponsored by the Brookings Institution on performance-based 
accountability (Elmore, Abelmann, & Fuhrman, 1995). The study was an analysis of two 
states— Mississippi and Kentucky— both of which instituted school accountability systems 
recently, but which developed very different systems. Based on their analysis, Elmore et al. 
(1995) concluded that states moving to performance accountability face five challenges: (a) 
making systems understandable and defensible, (b) designing and implementing fair systems, 
(c) focusing incentives for improvement where they will do the most good, (d) developing 
state capacity to maintain systems, and (e) creating a stable political environment for the 
survival of systems. 



What research is in progress in this area of KERA? 

Many individuals and research teams are continuing to collect or are planning to 
collect data relevant to assessment. Most importantly. Advanced Systems in Measurement 
and Evaluation (ASME), the contracted testing company, will continue to report technical data 
about assessment to the Commissioner of Education and to the State Board of Elementary and 
Secondary Education. The Kentucky State Department of Education may contract for 
additional studies to be performed on the assessment and accountability systems. 

It is likely that research on these systems will continue to be sponsored by the Office 
of Education Accountability (OEA) and the nonprofit Kentucky Institute for Education 
Research (KIER). Both organizations have an oversight role in monitoring implementation of 
Kentucky’s school reform, and both sponsored major evaluations of KIRIS and accountability 
published in 1995. In addition, the KIER sponsored statewide surveys of opinion about 
school reform (including assessment) have provided a valuable source of data on changes in 
attitudes about KERA. 

Several investigators have completed studies on assessment and accountability, and 
these may soon be published or released for public availability. 

Brian Gong and Edward Reidy of the Kentucky Department of Education have written 
a chapter to be published in the next yearbook of the National Society for the Study of 
Education (NSSE). The chapter deals with the relationships among accountability, 
assessment, and instruction in Kentucky’s school reform. 

In a comparative study of several states, Thomas Corcoran of the University of 
Pennsylvania (Center for Policy Research in Education— CPRE) has studied changes in science 
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and mathematics education in Kentucky. These changes are partly attributable to changes in 
assessment. A paper is scheduled to be published by Policy Studies Associates of 
Washington, D.C. 

Special education teachers in Kentucky have been surveyed on their attitudes toward 
assessment. The same survey was administered for three years (1992, 1993, and 1994). 
Results will be reported in 1996 by Jay McLoughlin of Cleveland State University, Joseph 
Petrosko of the University of Louisville, and several colleagues. 

Peter Winograd, Traci Bliss, a.id Eric Anderman at the University of Kentucky are 
engaged in a series of studies that explore the uses of financial incentives to reward teachers 
whose students score well on the statewide tests. These researchers are examining what 
effects offering teachers financial rewards based on students’ test scores has on the teachers’ 
sense of efficacy and perceptions about teaching. One of their papers was presented at the 
International Conference on Standards and Assessment in Raleigh, North Carolina in May 
1995. Data from approximately 70 teachers have been gathered and are in the process of 
being analyzed. The results will be available during the spring of 1996. 

A number of papers related to assessment have been produced by employees of the 
Kentucky Department of Education or of Advanced Systems for Measurement and Evaluation. 
Many of these were cited in KDE (1995). The reader should consult the latter volume to 
obtain author names, topics, and titles. However, two papers are too recent to have been cited 
in the KDE technical manual. Both relate to a critical aspect of KIRIS— the use of portfolios. 
One study is the final report of the mathematics portfolio scoring analysis, by Jan Broyles, 
Debbie Cox, Joann Mosier, and Sally Schneider (published by the Kentucky Department of 
Education). The second study, also available from KDE, is a study of three schools’ 
approaches to professional development and the Kentucky Writing Portfolio, by Amy 
Awbrey, Ann Bruce, Dyan Foree, Starr Lewis, and Caroline Pinne. 



Which questions or issues need further research? 

Two major evaluations of KIRIS published in 1995 raised many questions about the 
technical adequacy of the system. Consequently, assessment and accountability were very 
contentious issues in Kentucky during the past year. Given the role played by assessment and 
accountability within the school reform law, this is understandable. The authors of KERA put 
a high degree of emphasis on school outputs, and test score data are virtually the only means 
to measure these outputs. Issues like reliability and validity of assessment data are crucial 
concerns, because judgments about the overall success of school reform will be affected by 
the quality of assessment data. 

The state of Kentucky opted to build an entire assessment and accountability system 
from the ground up. In the words of Elmore et al. (1995), "Kentucky ... set out more or 
less intentionally to create one of the most advanced, and hence one of the most complicated, 
assessment and accountability systems in the country" (p. 17). In 1990, when the school 
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reform law was passed, the problematic nature of this enterprise was known by relatively few 
people. In 1995, it is known by many citizens in the state. 

Changes are already planned for KIRIS, because the current contract with the testing 
company that now operates the system, Advanced Systems for Measurement and Evaluation, 
is due to expire in 1996. However even a revised KIRIS will face certain issues that are 
worthy of research, as long as the major testing and accountability provisions of the law stay 
in place. 

Below are several research areas that need to be examined. 

% 

1. Research is needed to maximize fairness when categorizing students (e.g., Novice, 

Apprentice) and when putting schools in outcome categories (e.g., Reward, In Crisis). 

This was a major concern of both major evaluations completed in 1995: by the OEA 
measurement panel and by the evaluation team from Western Michigan University. 
There are really many issues subsumed under this topic; however only two will be 
mentioned. 

One issue pertains to the methods of estimating the reliability of KIRIS scores. The 
formulas used to calculate test score reliability must be defensible, in light of what is 
known by experts in generalizability theory. This is a challenge, because KIRIS has 
many complex features. For example, scores from a school are based on both 
common test items (that every student completes) and matrix sampled items (that only 
some students complete). Furthermore, there are several assessment formats (open- 
response items, portfolio scores). Psychometric research, or at least consultation with 
several experts to obtain a consensus opinion, may be needed to arrive at the most 
realistic formulas to calculate reliability of KIRIS tests. Such research and 
development efforts should be a high priority, because continued questioning of 
methods used to estimate test score reliability will undermine acceptance of the data. 
Without the use of defensible reliability procedures, there will continue to be 
skepticism about test scores on individual students and about accountability judgments 
made on schools. 

Another area where research could be done is on the topic of decision accuracy of 
accountability judgments. Every educational test has some measurement error; hence, 
there will always be errors in using average test score data to assign schools to various 
ordinal categories (e.g., Reward, Improvement). The question is, how many decision 
errors are too many? What if 33% of the schools declared "in crisis" by KIRIS data 
were put there erroneously, because of measurement error? Is 33% the best that can 
be expected? Policy research would be helpful to provide a context for the 
interpretation of such data. 

For example, how many decision errors would be made if schools were categorized in 
a state using data from a nationally standardized test, like the Comprehensive Tests of 
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Basic Skills (CTBS)? (Data from outside Kentucky would probably best be used to 
perform such studies.) It is true that a policy judgment is eventually required to judge 
the adequacy of accountability decisions. Someone (the State Board of Education?. . . 
the general public?) has to ultimately decide what percentage of erroneously classified 
schools the state can tolerate, whether 33% or some other percentage. But currently 
there is no empirical context for such decisions. Research would be valuable to 
inform policy decisions on what error rates can be expected when using various types 
of test score data to assign schools to ordinal categories of performance. 

2. A program of research should be initiated on the most effective methods of 
professional development to help teachers integrate assessment and instruction. 

The implementation study of performance assessment by Matthews (1995) showed 
considerable variability among classrooms in the degree to which teachers have 
changed instruction because of KIRIS. Even when teachers were cognizant of KIRIS 
and attempting to prepare students for testing, these efforts were often disconnected 
from everyday instruction. Research and development are needed to identify the best 
ways of helping teachers embed performance assessment in regular instruction. 

3. Studies should be done on the best means of using portfolios in assessment and 
instruction. 

In .ome ways, this is a special version of the second point just discussed. However, it 
is identified separately because portfolios have had an important (and controversial) 
role in KIRIS. It is uncertain whether portfolio scores will continue to be used as part 
of the school accountability formula. However, portfolios will probably continue to be 
used in some fashion in Kentucky. The question arises, what are the optimal 
instructional uses of portfolios? Can they be made a natural part of instruction? How 
much does the subject matter of the portfolio affect its usefulness in instruction? For 
example, can some instructional strategies successfully used by writing teachers with 
their portfolios also be used by mathematics teachers with mathematics portfolios? 

4. Research should be performed on how best to use KIRIS to evaluate the local 
curriculum. 

A variety of KIRIS reports is sent to schools. How can these data be used as part of 
the process of curriculum evaluation in a school or school district? Are the kinds of 
information that will be available on a revised version of KIRIS likely to help schools 
improve their school-wide instruction in the various subject areas? 

5. Studies should be done on how teachers have been affected by receiving, or not 
receiving, reward money during the first accountability cycle of KERA. 

Kentucky educators in "reward schools" received approximately 26 million dollars in 
1995. Has this money had any positive effects? Ostensibly, rewards were earned 
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because of superior instruction. Are rewarded educators continuing to do what got 
them a reward? Has not getting money negatively affected teachers in "non-reward" 
schools? Have teachers who were not rewarded changed their teaching? 
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KERA PRESCHOOL PROGRAMS FOR 
AT-RISK 4-YEAR-OLD CHILDREN AND 3- AND 4-YEAR-OLD 
CHILDREN WITH DISABILITIES 

Mary Louise Hemmeter 
University of Kentucky 



What does the law or regulation mandate? 

Section 157.3175 of the Kentucky Revised Statutes authorizes the implementation of 
preschool education programs for four year old children considered to be at-risk. Children 
are considered "at-risk", and thus eligible for the KERA Preschool Program, if they qualify 
for free lunch based on the federal free lunch guidelines. Section 157.226 of the Kentucky 
Revised Statutes also authorizes preschool education programs and related services for 
preschool (three- and four-year-old) children with disabilities who are eligible for services 
under the federal law, P.L. 99-457. 

The regulations indicate that the intent of the law is not to supplant existing services 
but to work cooperatively with service delivery systems (e.g., Head Start, Private Day Care, 
Private Early Intervention Programs) to provide comprehensive services to young children 
who are at-risk and young children with disabilities. This provides districts with the 
opportunity to contract with existing agencies to provide services for eligible preschool 
children. Additional requirements of the preschool programs are the provision of the 
following services: a) parent involvement activities that take into consideration the variety of 
family structures that will be involved; b) a minimum of two home visits per child per year; 
c) collaboration with medical, health, mental health, and social service agencies; and d) child 
development screening. 



Review of the Research to date. 

To date, only a few studies of the KERA Preschool Programs have been implemented. 
The largest study is a statewide longitudinal evaluation of the programs that is being 
conducted by a group of researchers representing the Colleges of Education and Human 
Environmental Sciences at the University of Kentucky (Peck, Bridge, Townley, & Hemmeter, 
1992; Bridge, Townley, & Hemmeter, 1993; Bridge, Townley, Hemmeter & de Mesquita, 
1994, 1995). This project has focused on examining three primary issues: a) the quality of 
the programs; b) the effects of the programs on children in terms of overall development, 
social skills, and early literacy skills; and c) consumer satisfaction with the Preschool 
Programs. In addition, several questions related to the KERA Preschool Programs were 
included on a Statewide Education Reform Survey conducted by the Kentucky Institute for 
Education Reform in September of 1995. This survey asked principals, teachers, parents and 
the general public to rate their perception of how well the KERA Preschool Programs were 
doing. 
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The review of the research will be divided into four sections. The first section will 
provide an overview of the extent to which the KERA Preschool Programs are being 
implemented. The implementation data that will be reported in this section were obtained 
from the Kentucky Department of Education. The second section will address child outcomes 
associated with participation in the KERA Preschool Programs, and will focus primarily on 
the longitudinal evaluation being conducted by the University of Kentucky. The third section 
will address consumer satisfaction with the KERA Preschool Programs and family 
involvement in the KERA Programs and will focus on data from the longitudinal evaluation 
as well as data from the Statewide Education Reform Survey conducted by the Kentucky 
Institute for Education Reform. The fourth section will address the quality of the KERA 
Preschool Programs. 

Program Quality 

During the 1991-1992 evaluation observations were conducted to determine whether 
the Preschool Programs were being implemented as recommended by early childhood experts. 
Half of the KERA Preschool Classrooms involved in the Third Party Evaluation were 
observed using the Early Childhood Environment Rating Scale - ECERS (Harms, Clifford, & 
Bailey, 1987). The ECERS was used again during the 1993-1994 evaluation to determine the 
extent to which the quality of the programs had improved. 

The mean ECERS scores improved from 4.5 (on a 7-point scale) in 1992 to 5.1 in 
1994. Overall, the 1993-1994 classrooms were above average in all areas except two. They 
scored below the mean on items related to cultural awareness and provision of space for 
children to be alone. The individual classrooms ranged in quality from low to excellent with 
over 50% of the programs falling in the range from 4.75 to 5.75. During the 1993-1994 
evaluation, six items designed by the authors of the ECERS related specifically to children 
with disabilities were administered in all classrooms in which there were children with 
disabilities. All of the items were rated slightly above the mean. 



To what extent has the program been implemented as intended by KERA? 

During the 1994-1995 school year, a total of 15,374 preschoolers were served in or 
through the KERA Preschool Program. This represents approximately 7,827 children without 
disabilities and 7,547 children with disabilities. Of the children with disabilities approximately 
49% were children with speech and language delays only, 46% were children with 
developmental delays, and 5% were children with severe disabilities. Most districts reported 
an increase of approximately 5% in their enrollment figures from the 1993-94 school year. 

An additional 13,212 children were served with federal dollars through Head Start. The cost 
of the programs during the 1993-94 school year was approximately 38.5 million dollars. 

In addition to the numbers of children served, data are available on the number of 
children who received other services associated with the KERA Preschool Programs. All 
children received at least one meal per day, while 53% of the children received two meals per 
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day. Half of the children involved in the programs had parents who volunteered in the 
program. Ninety seven percent of the participants were immunized and had a health 
screening, and 87% of the children who needed health follow-up or treatment received this. 



What have been the effects of the program on stakeholders? 

The Third Party Evaluation of the KERA Preschool Programs has been in progress 
since the fall of 1991 and has focused primarily on child outcomes, program quality, and 
stakeholder satisfaction. Major findings of the evaluation over the last four years indicate 
that: 

% 

1. Children who attend KERA preschool programs make significant gains during the year 
they are in preschool. When compared to an income-eligible group of children who 
did not attend the KERA Preschool Programs, the KERA participants do significantly 
better. 

2. Longitudinal follow-up of children who attended the KERA Preschool Programs in 

1990- 1994 indicate that they do as well as or better in many areas during their primary 
years than a random group of their peers who represent a range of income levels and 
who attended a variety of preschool programs during their preschool years. 

A more detailed description of the findings from each year of the Third Party 
Evaluation is provided below. 

1991- 1992 Evaluation (Peck, Bridge, Townley, & Hemmeter, 1992) 

A stratified (urban versus rural, SES level of district, location in state) random 
sampling procedure was used to select 36 school districts that would participate in the 
evaluation. Within each of the 36 districts, two sites were identified in which children would 
be tested. Four hundred and thirty two children attending the KERA Preschool Programs 
during the 1991-1992 school year were tested. This number included a representative sample 
of children at-risk and children with disabilities. 

In terms of developmental skills, there were only two domains on the Battelle 
Developmental Screening Inventory on which more than 50% of the children scored above 
the cutoff. Teachers rated the majority of the program children as at or above average on 
social skills as measured by the Social Skills Questionnaire regardless of whether the children 
were at-risk or had disabilities. In order to assess the effects of participation in the KERA 
Preschool Programs on previous participants, two additional groups of children were tested. 
Primary 1 children who had attended the KERA Preschool Programs during the previous year 
were tested (Cohort 1) along with a second group of Primary 1 children who had not attended 
the Preschool Programs the previous year to serve as a comparison group. The findings with 
the Primary 1 children indicated that children who had attended the KERA Preschool 
Programs scored significantly higher on the Battelle than the children who had not attended 




101 



14 £ 




the Preschool Programs the previous year. However, on the social skills questionnaire, the 
children who had attended the KERA Programs scored significantly lower than the children 
who had not attended the Preschool Programs the previous year. Since the comparison 
children represented a range of income levels and preschool experiences, it is encouraging 
that the KERA participants outscored a random sample of their peers on the Battelle. 

1992-1993 Evaluation (Bridge, Hemmeter, & Townley, 1993) 

The 1992-1993 evaluation data were collected in the same districts that were used 
during the 1991-1992 evaluation. In addition to testing KERA Preschool participants and a 
comparison group of eligible nonparticipants, 543 of the children tested during the 1991-1992 
evaluation were retested. An additional control group of 137 children was identified and 
tested for 1991-1992 preschool children (Cohort 2). For the 1992-1993 evaluation, a battery 
of early literacy measures was added to the battery of instruments that were administered to 
the children during the 1991-1992 evaluation. 

Three hundred and twenty program children and 106 comparison children from 
Cohort 3 were tested. The comparison children included income-eligible children who had 
been in other programs and income-eligible children who had not been in other programs. 

The mean Battelle scores for the program children were higher than the scores of the 
comparison children across all domains. In addition, the scores for the program children were 
significantly higher than those of the comparison children in all domains except gross motor 
and expressive communication. On the three early literacy measures, the program children 
scored higher than the comparison children, although the differences were not statistically 
significant. On the Harter, the program children perceived themselves to be significantly 
more accepted by their peers and more competent on cognitive tasks than did the comparison 
children. 

Two hundred and eighty six program children and 82 comparison children from 
Cohort 2 were tested. The program children were randomly selected from the children who 
had been tested the previous year. However, no comparison children had been tested the 
previous year. Therefore, a comparison group of children had to be identified. The 
comparison children were randomly selected from the primary classrooms of the children who 
had been tested the previous year. They included children who were income eligible and 
children who were not, as well as children who had been in programs during their preschool 
year and children who had not. The program children scored higher than the comparison 
children on five of the Battelle domains. The program children also scored higher on two of 
the three early literacy measures. However, the differences on the Battelle and the early 
literacy measures were not statistically significant. The teachers rated the program children 
significantly higher on the academic competence area of the Social Skills Questionnaire. This 
indicates that the teachers perceive the children who attended the KERA Preschool Programs 
to have better overall academic performance, more overall motivation, and more parental 
encouragement than the children who did not attend the KERA Preschool Programs. 
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One hundred and nineteen program children and 30 comparison children from Cohort 
1 were located and tested. These children were randomly selected from the children who had 
been tested during the previous evaluation. The comparison children represented the same 
range of children as the comparison children for Cohort 2. The program children scored 
higher on six of the Battelle domains than the comparison children. The scores for the 
program children were significantly higher on the expressive communication domain than 
those of the comparison children. The program children also scored significantly higher than 
the comparison children on the Letter Recognition Test and the Book Handling Knowledge 
Test. The teachers rated the program children significantly higher on the academic 
competence area of the Social Skills Ques onnaire, again indicating that the teachers perceive 
the children who attended the KERA Preschool Programs to have better overall academic 
performance, more overall motivation, and more parental encouragement than the children 
who did not attend the KERA Preschool Programs. 

In summary, children who had attended the KERA Preschool Programs generally had 
higher scores than the comparison children. The most significant results were found for 
Cohort 3 who were the children who were in the KERA Preschool Programs during the 
1992-1993 school year. However, the comparison children for Cohorts 1 and 2 included a 
random sample of all children, not just income-eligible children as was true for the Cohort 3 
comparison children. Therefore, the expectation is that the program children in Cohorts 1 
and 2 should score as well as the comparison children. The findings indicated that when 
there were significant differences for Cohort 1 and 2, they favored the program children. 
Therefore, the program children in Cohorts 1 and 2 were doing as well as, and in many cases, 
better than a random sample of their peers. 

1993-1994 Evaluation ( Bridge , Townley, Hemmeter, & deMesquita, 1994). 

The third year of the Evaluation of the Preschool Programs extended the previous 
evaluations in two important ways. First, a group of KERA Preschool Children (Cohort 4) 
and a comparison group of eligible children not attending the KERA Preschool Programs 
were tested at the beginning and the end of the school year. This made it possible to 
document the short-term effects of the program. Second, all of the programs in which 
children in Cohort 4 were tested were observed in order to obtain data on program quality. 
These data were used to examine the relationship between program quality and child 
outcomes. In addition to these activities, children from the three previous cohorts were 
located, and a random sample of children were tested. A slightly different sampling 
procedure was used this year to identify the KERA Preschool Participants who would be 
tested as part of the evaluation. Children were selected from the same districts that had been 
used during the previous years, but in previous years, the same number of children were 
selected from each district. However, for the 1993-1994 evaluation, the number of children 
tested in each district was weighted, based on the relative size of the district compared to the 
size of the other districts from which children were tested. 

Approximately 1,300 current and former KERA preschool participants and comparison 
children randomly were assessed on a variety of measures designed to measure their 
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developmental skills, social skills, and early literacy skills (See Table 1). Not only did the 
children who participated in the program in 1993-1994 demonstrate significant gains during .the 
program, but also the children who had participated in the program in previous years (1990-1993) 
demonstrated achievement levels similar to other children in their regular classrooms, many of 
whom came from more advantaged backgrounds. The 1993-1994 KERA preschool participants 
made statistically significant gains from pretest to posttest in many developmental skills necessary 
for school success: adaptive (self-help), personal social, motor, communication, and cognitive 
skills as well as in their knowledge of the alphabet and print concepts. Their teachers and 
parents judged them to have more positive social skills and fewer behavioral problems after 
participating in the preschool program. When compared to a group of income-eligible peers who 
had not attended the KERA preschool prograrn, but three-fourths of whom had attended other 
child care programs, the KERA preschoolers significantly outscored their peers in adaptive 
(self-help) skills and fine motor skills. Both teachers and parents rated their social skills higher, 
and teachers reported that they had fewer behavior problems. 

Follow-up tests of former participants revealed that when KERA preschoolers were 
compared to a random sample of classmates of various income and ability levels, they scored as 
well as or better than their classmates in many developmental skills, social skills, and early 
literacy skills. The only exception was in the social skills of the children who had participated 
in the first year of the preschool and who were in the third year of primary school. These 
children were rated by their teachers as having less positive social skills, more problem 
behaviors, and lower academic competence. 

Results for children with disabilities also showed positive trends. Children in the 
developmentally at-risk group made particularly impressive gains that exceeded the gains of the 
economically at-risk children who had no disabilities. However, the small number of children 
in this group (n=8) necessitates that the results be interpreted cautiously. The children with 
developmental delays (n=47) and the children with speech-language problems (n=75) gained at 
a rate equal to the economically at-risk children. Even the children with severe disabilities (n=3) 
made gains that nearly equaled the economically at-risk children. 



1994-1995 Evaluation ( Bridge , Townley, Hemmeter, & deMesquita, 1995). 



During the 1994-1995 evaluation, 329 KERA Preschool Participants and 53 eligible 
children who were not attending the KERA Preschool Programs were tested. These children 
were selected using the same procedures described in the 1993-1994 evaluation. Results 
indicated that the rate of overall development was greater for the KERA Preschool Participants 
than for the children in the comparison group. Their projected gains were significantly greater 
in the area of fine motor and on the overall score on the Battelle Developmental Inventory. In 
addition, teachers rated the participants as having significantly greater social skills and 
significantly fewer problem behaviors than the children in the comparison group. While the 
participants scores were higher on all measures of early literacy than the scores of the comparison 
group, only one measure (writing lower case letters) was significantly higher. In addition to 
testing the children who were considered to be at-risk, a group of children with disabilities was 
tested. Positive gains were observed across all three groups of children with disabilities (i.e., 



148 ' 



o 



104 



speech and language, developmental delays, severe disabilities). The gains were most consistent 
for the children with developmental delays and speech and language delays as compared to .the 
children with severe disabilities. However, it is clear that children with disabilities are making 
the most gains in those areas in which they have the greatest delays, suggesting that the program 
is addressing the unique needs of each child. While there was no comparison group for these 
children, the finding that they are gaining one month in some areas per month in intervention is 
important, particularly given that their rate of development prior to the program was significantly 
slower. 



Children from the previous four cohorts were located. Parent and teacher social skills 
questionnaires, and teacher surveys of student progress and motivation were sent to the parents 
and teachers of all students from these cohorts. For Cohort 4 (n= 137), the teachers indicated that 
approximately three-fourths of the children were achieving as well as or better than the rest of 
the children in their class in terms of their attainment of Kentucky’s six learning goals. The 
lowest rated area was in Thinking and Solving Problems with 68% of the children judged to be 
doing as well as or better than most of the children in their class. In the highest rated area, 78% 
of the children were rated as doing as well as or better than the rest of their class in the area of 
Becoming Self Sufficient. Teachers were also asked to rate the former KERA preschool 
children’s performance in various curriculum areas. At least two-thirds of the children were rated 
as doing as well as or better than most of their peers in all curriculum areas. 

The patterns observed for Cohort 4 were also observed for Cohort 3. When rating the 
children’s attainment of Kentucky’s six learning goals, the teachers judged two-thirds or more 
of the children in Cohort 3 (n=123) to be doing as well as or better than most of their peers. 
Ratings were similar for teachers’ judgments of children’s performance in various areas of the 
primary curriculum. Over half of the children were rated as doing as well as or better than their 
peers. The children’s progress in reading, writing and math was rated lower than their progress 
in social studies, science, art, music, and physical education. The teachers’ ratings of the former 
participants were then compared to the same age comparison children identified earlier. The 
results indicated that the program children were doing as well as or better in all areas than the 
comparison children, although the differences were not statistically significant. 

Again, the patterns observed for children in Cohorts 3 and 4 were observed for Cohorts 1 
and 2. Teachers rated the academic progress and expectations for future success of the former 
participants to be the same or better than most of their peers. When rating the children’s 
attainment of Kentucky’s six learning goals, the teachers judged two-thirds or more of the 
children in Cohorts 2 (n=70) and 1 (n=50) to be doing as well as or better than most of their 
peers. When teachers’ ratings of the participants were compared to teachers’ ratings of the 
nonparticipants, the teachers rated the participants in Cohort 2 as doing as well as or better than 
the comparison children. Again, the participants were rated higher or equal to the nonparticipants 
in most areas although the differences were not significant. While the differences observed in 
Cohort 1 favored the comparison children, these differences were not significant. 

Finally, the results of the Social Skills Rating System completed by teachers and parents 
indicated that the KERA participants generally do as well as or better than the comparison 
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children. The findings across all cohorts indicate that the KERA participants are generally within 
the average range for their age and grade levels. These findings are also true across the three 
areas of social skills, problem behaviors and academic competence. Teacher and parent ratings 
are generally in agreement in their views of these children as socially more similar to their peers 
than different. Over the last few years of the evaluation, their results of the social skills ratings 
have suggested a positive trend indicating that children who participated in KERA Preschool 
Programs are rated by their teachers as no different than their peers in the primary levels one, 
two and three. These findings are positive in that they suggest that children considered to be 
at-risk when they enter preschool are doing as well as a random group of their peers, including 
children from families of all income levels. However, for the last two years, teachers have 
consistently reported significant differences in both social skills and problem behaviors in favor 
of the comparison children for the oldest cohort. This raises the question about whether gains 
maintain over time. However, it is possible that the quality of the preschool programs during the 
first two year of implementation was not adequate for affecting changes that would maintain over 
time. The ongoing longitudinal study will provide data that will address this issue. 



What are the attitudes and perceptions of stakeholders toward the program? 

Consumer Satisfaction and Family Involvement 

This section of the review will address consumer satisfaction with the KERA Preschool 
Programs and family involvement in the KERA Programs and will focus on data from the 
longitudinal evaluation as well as on data from the Statewide Education Reform Survey 
conducted by the Kentucky Institute for Education Reform. During the 1992-1993 Third Party 
Evaluation, surveys related to consumer satisfaction and family involvement were sent to teachers 
and parents. In addition, surveys were sent to community agencies involved in providing services 
to the population of children who participate in the KERA Preschool Programs. Focus groups 
were conducted with teachers, administrators, and community agencies. During the 1994-1995 
evaluation, surveys on parent satisfaction and parent involvement were sent to parents. Finally, 
the Kentucky Institute for Education Reform conducted a statewide survey on education reform 
that included questions related to satisfaction with the Preschool Programs. 

In the Spring of 1993, 24 focus groups were conducted across the state. In each of six 
areas of the state, separate focus groups were held for teachers, administrators, and 
community agency representatives and focused primarily on attitudes toward the KERA 
Preschool Programs. Due to poor attendance at many of the focus groups, findings must be 
interpreted with caution. A total of thirteen teachers attended the focus groups and were 
generally positive about the programs and the effects of the programs on children. They 
consistently indicated that the most significant impact of the programs was on preparing 
children for transition to kindergarten. Their primary concern was the lack of time they had 
to implement the parent involvement component of the program. Four administrators 
attended the focus groups. They reported two primary benefits of the Preschool Programs. 
First, they indicated that the early health screening identified children with disabilities earlier, 
thus they received services earlier than they might have in the absence of the KERA 
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Preschool Programs. Second, they indicated that the programs helped acclimate the children 
to school prior to kindergarten. They reported that they liked the parent involvement 
component but that it was not fully developed. Finally, the principals felt that the programs 
should be available to ail children regardless of income level or disability. Twenty-two 
representatives from a variety of community agencies attended the focus groups. The primary 
finding from the focus groups for community agency representatives was related to 
collaboration. Agency representatives expressed concerns related, to the duplication of 
services, primarily as it related to Head Start. Collaboration appeared to be working better in 
rural areas than it was in urban areas. 

During the spring of 1993, surveys were sent to parents, teachers and community 
agency representatives. The teacher survey included items related to the types of parent 
involvement activities they offered to families, the types of agencies to which they referred 
families, barriers to parent involvement activities and referral activities, factors related to 
serving children with disabilities, and the extent to which they felt that the programs were 
helpful to children and families. The parent survey was similar to the teacher survey in that 
it included items related to the parent involvement component of the KERA Preschool 
Programs, items related to the types of community agencies to which they had been referred 
by the Preschool Program, and the extent to which the programs were helpful to their children 
and their families. Finally, the community agency survey requested information on the types 
of contact they had had with the Preschool Programs and their perception of their relationship 
to the Preschool Programs. The results of the teacher survey indicated that teachers were 
offering a wide variety of activities to parents. The most frequent activities they reported 
were home visits, phone calls, field trips, home activities, conferences and parent newsletters. 
Over half of the respondents indicated that they felt that their offerings are adequate. Of 
those who did not feel that their offerings are adequate, 40% indicated that they would offer 
more if they had more time. The parents reported that many activities were offered to them 
by the programs. The highest rated offerings were similar to those indicated by the teachers. 
The most common barrier for parents was scheduling conflicts. Teachers, community agency 
representatives and parents reported that families were being referred to community agencies. 
The most common referrals appeared to be to Head Start, dental services, and other health 
agencies. Most of the teachers felt that they were making adequate referrals and that one of 
the primary reasons they did not make more was that the parents did not want to be referred. 
This was consistent with the parent responses in that less than a third of the parents who were 
referred actually sought the services. The parents indicated that the reason they were not 
using the services was that they did not feel that they needed them. The community agencies 
rated their relationship with the Preschool Programs as a 2.32 on a 5-point scale with 1 being 
the highest. They indicated that they did not feel that the referrals were putting a strain on 
their programs. 

Parents and teachers were asked to rate the extent to which they viewed the programs 
as being helpful to children and families. The average rating for the parents and teachers 
with regard to the value of the program for children was a 1 .22 (on a 5-point scale with 1 
being the highest). The average rating for the parents with regard to the value of the program 
to families was 1.57 while the average rating for the teachers was 1.58. When asked what 



O 

ERIC 



107 



151 



they would do to change the program, the most common response of the parents was to make 
it available to all children. Finally, teachers were asked questions related to serving children 
with disabilities. The findings indicated that teachers were typically involved in both the 
development and implementation of children’s Individualized Education Plans. Many teachers 
indicated that they had the assistance necessary for serving most children with disabilities. 
However, they indicated some concern about serving children with severe disabilities and 
behavior problems. They indicated the need for more assistance in order to have additional 
time to work individually with children with severe disabilities and behavior problems. 

During Spring of 1995, the survey used during the previous evaluation was expanded. 
The questions about family involvement activities remained the same, but several questions 
related to parent satisfaction with various aspects of the Preschool Programs were added. As 
was found during the previous survey, the 1995 survey found that a wide variety of activities 
was being offered to families, and that families were participating in a variety of activities. 
However, it was noted that there was a discrepancy in the frequency with which activities 
were offered and the frequency with which parents were choosing to be involved in the 
activities. While parents identified particular barriers to their participation, the results suggest 
the need for involving parents in the identification of parent involvement activities as a 
strategy for ensuring that offered activities are consistent with the needs and desires of the 
families. Parents reported high levels of satisfaction with the programs and with the effects 
of the programs on their children’s development. Over 95% of the parents indicated that the 
program was helpful to their child and that they felt comfortable going to talk to their child’s 
teacher. On 9 of the 13 questions, 90% of the parents who responded indicated that they 
agreed or strongly agreed in terms of their satisfaction with various aspects of the program. 

Finally, the KIER statewide education reform survey provided additional information 
on consumer satisfaction with the Preschool Programs. Of the 214 principals that were 
interviewed, 75.6% of them indicated that they felt the Preschool Programs were working 
well. Only 2.7% indicated that the Preschool Programs were not working well. These figures 
are significant in that they reflect the most positive ratings of all programs about which the 
principals were surveyed. The results were similar when teachers were asked to rate how 
well certain KERA initiatives were working. Of the 609 teachers surveyed, 65% of them 
indicated the Preschool Programs were working well while only 6.6% of them felt the 
programs were not working well. Twenty-eight of them indicated they were undecided. This 
may be due to unfamiliarity with the Preschool Programs. Similar patterns were observed 
with the school council parents who were interviewed. Of the 100 parents interviewed, 

67.7% of them indicated that the Preschools were working well while only 6.5% felt they 
were not working well. The results were quite different when a group of public school 
parents were interviewed. Only 46.8% of them felt the Preschool programs were working 
well, while 43.8% felt they were working poorly. The general public also indicated some 
discontent with the Preschool Programs. Of the 214 citizens interviewed, only 38.4% of them 
felt the Preschool Programs were working well. This could reflect lack of knowledge of the 
programs or it could reflect the feeling that the Preschool Programs should be available for all 
children. When educators who indicated that the program was not working well were asked 
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about factors contributing to the program not working well, the most common response was 
that the Preschool Program was poorly designed. 



What research is in progress in this area of KERA? 

The fifth year of the Third Party Evaluation of the KERA Preschool Programs was 
initiated in August (Bridge, Townley, Hemmeter, & de Mesquita, in progress). The focus of 
the 1995-1996 evaluation will be on evaluating child outcomes and investigating the 
relationship between program quality and child outcomes. In addition to collecting child 
outcome data on six cohorts of children, both quantitative and qualitative data are being 
collected on program implementation and quality in the 24 preschool classrooms in which 
children are being tested. 

A second series of studies that is being implemented relates to both the Preschool and 
the Primary Programs. A group of researchers from the University of Kentucky (Hemmeter 
and Schuster), Allegheny Singer Research Institute in Pittsburgh and the University of North 
Carolina at Chapel Hill is investigating strategies for embedding effective early childhood 
practices (developmentally appropriate practices, family centered services, integrated therapy) 
into primary programs. This project is funded by a grant from the federal government. Over 
the course of the next five years, over 15 studies will be conducted. Approximately half of 
those studies will be conducted in Kentucky. The goal of the first two years is to identify 
barriers to implementing these practices in ungraded primary programs. Observations, 
interviews, focus groups, and surveys will be used to ensure that information on barriers is 
obtained from all relevant professionals as well as families. The third and fourth years will 
focus on evaluating strategies for overcoming these barriers, and the fifth year will focus on 
strategies for disseminating the information on a local, regional and national basis. 



Which questions or issues need further research? 

There are several important research questions that have not been addressed or 
answered in the research reviewed above. While it is not possible or practical to list all of 
those questions here, there are several questions which, because of the goals of the program 
and the opportunity provided by the program, are most critical at this point. 

From a policy perspective, one of the highest priority research issues relates to the 
extent to which the preschool programs are being delivered collaboratively rather than 
supplanting existing services and the relative effects of the different program for young 
children. If services are being duplicated, the issue is a financial one. If KERA Preschool 
Programs are supplanting existing programs, potential for conflict within the field of early 
childhood education is high. First, there must be an evaluation of the extent to which 
services are being supplanted or duplicated. Second, there must be an evaluation of the 
relative effectiveness of programs provided by school districts and programs provided by 
existing agencies. Data on the relative effectiveness of the different program should be used 
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in determining the most appropriate service delivery model. In addition, consumer opinion 
and satisfaction should be evaluated. 

A second critical question relates to the relationship between program quality and 
child outcomes. The Third Party Evaluation has addressed and continues to address the issues 
of child outcomes and program quality; no data are yet available on the relationship between 
the two. While the Third Party Evaluation has found significant child outcomes as a result of 
participation in the KERA Preschool Programs, the children who have participated in the 
evaluation have attended KERA Preschool Programs that represent both high and low quality. 
Thus, a more fine-grained analysis is needed to identify quality indicators and program 
components that are most likely to lead to better child outcomes. 

The relationship between teacher training and preparation and program quality is 
another area that poses a critical research question. This relationship is multifaceted and as 
such, will require that multiple questions be asked. What kind of training results in the 
implementation of high quality programs? Are teachers who have a certain kind of training 
more likely to have higher quality programs than teachers who have a different type of 
training? Do teachers with training in one area have students who make more positive gains? 
While these are important and reasonable questions, a more practical question related to this 
area is, "What is the relationship between teacher identified training needs and the actual 
training teachers are getting in Kentucky?" While the quality of preservice training is critical, 
it is important that inservice training be investigated as well. 

Another important question relates to the extent to which gains made during preschool 
maintain during the primary years and beyond. Previous research on preschool programs 
generally indicates that gains related to specific curriculum areas begin to fade out as the 
child moves through the primary grades. An important related question is, "What is the 
relationship between the extent to which children’s gains made during preschool are 
maintained and the quality of the primary programs which they attend when they leave 
preschool?" Intuitively, we should not expect that gains will be maintained if the quality of 
the child’s education beyond preschool is poor. On the other hand, it is possible that children 
learn generic skills during preschool that help them adapt to poor quality programs during 
later years. Clearly, the answers to these issues must be addressed. 

Finally, the KERJ^-Pfe school Programs provide a unique opportunity to evaluate the 
effects of inclusive preschool programs on children with and without disabilities. One of the 
barriers to research on inclusion in preschool over the last ten years is lack of preschool 
programs that are truly inclusive. There are many research questions that should be addressed 
in this context. How does participation in inclusive preschool programs affect children’s 
attitudes about people with disabilities? Does participation in inclusive preschool programs 
affect children’s behavior towards people with disabilities? What are the effects of inclusion 
on families? In addition to questions related to att nudes, there are questions related to the 
effects of inclusion on children with disabilities. To what extent are the IEP goals and 
objectives of children with disabilities embedded into routine activities in the classroom, and 
what are the effects of this embedded instruction on children’s acquisition of new skills? 
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Because inclusion is such a prominent issue on a national level, much research is needed, and 
the KERA Preschool Programs provide a unique context for conducting that research. 
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PRIMARY PROGRAM 

Ellen McIntyre and Diane W. Kyle 
University of Louisville 



What does the law or regulation mandate? 

KRS 158.030(2) states, 

Primary School Program is that part of the elementary school program in which 
children are enrolled from the time they begin school until they are ready to 
enter fourth grade. Notwithstanding any statute to the contrary, successful 
completion of the primary school program shall be a prerequisite for a child’s 
entrance into fourth grade. The State Board of Elementary and Secondary 
Education shall establish, by regulation, methods of verifying successful 
completion of the primary school program pursuant to the goals of education as 
described in KRS 185.6451. 

Furthermore, KRS 158.030(1) states that "...Any child who is five (5) years of age by 
October 1, may enter a primary school program as defined in subsection (2)...," and KRS 
157.320(7) clarifies that "...Kindergarten is the entry level of the primary program." 

Subsequently, House Bill 187 indicates 

A school council established pursuant to KRS 160.345 or if none exists, a 
school may determine, based on individual student needs, that implementing 
multi-age and multi-ability classrooms need not apply for every grouping of 
students for every activity throughout the entire day.. .and will allow for 
grouping of students attending their first year of school when determined to be 
developmentally appropriate. (Kentucky Department of Education, 1994, p. 24) 

The State Board of Elementary and Secondary Education developed a position 
statement on multi-age/multi-ahi!itv grouping in the primary program which clarified that, 

"The focus on developmental] iropriate practices and individual student needs in making 
grouping decisions will most < i result in multi-age and multi-ability classrooms" 

(Kentucky Department of Education, 1994 p. 24). They further declared that the bill did not 
permit schools "to decide not to implement an appropriate primary program, which continues 
to be required by law to include multi-age/multi-ability grouping" (Kentucky Department of 
Education, 1994, p. 24). 

The philosophy of the nongraded primary program mandated by KERA stems from the 
National Association for the Education of Young Children’s most recent position statement on 
developmentally appropriate practices for children of ages 5-8 (NAEYC, 1988). This 
statement reflects "the most current knowledge of teaching and learning as derived from 





theory, research, and practice" (p.64). It recognizes the need for teachers to teach to the 
whole child — his/her physical, social, emotional, and cognitive dimensions -- within an 
integrated curriculum in which children are engaged in active, hands-on activities rather than 
passive, paper-and-pencil activities only. It recognizes the importance o.:’ social interaction 
around meaningful, relevant topics of interest to the children. It also emphasizes the 
importance of a match between the capabilities and experiences of young children and the 
instructional opportunities provided them. 

The position statement of the Kentucky primary school program (Kentucky 
Department of Education, 1994) identifies these critical attributes of the program: 

t 

1. Developmental^ appropriate practices; 

2. Multi-age/multi-ability groupings; 

3. Continuous progress; 

4. Authentic assessment; 

5. Qualitative reporting methods; 

6. Professional teamwork; and 

7. Positive parent involvement. 



Review of Research to date. 

To what extent has the program been implemented as intended by KERA? 

Several studies have examined the development and implementation of the KERA 
Primary Programs. These have included large-scale "snapshot" studies leading to conclusions 
about general characteristics of primary programs across the state and long-term, in-depth 
studies leading to conclusions and explanations about specific decisions, practices, and 
influences within primary programs. 

Studies of the progress of primary programs (Appalachia Educational Laboratory 
[AEL], 1993; Bridge, 1993, 1994, 1995; Bruno & Johnson, 1994; Chmiel, 1992; Coe, 1995; 
Craig, in progress; Gregory, Kyle, Moore, & Wheatley, 1995; Hovda, Davis, & Monarch, 
1995; Kyle & McIntyre, 1993; McGee, 1994; McIntyre, Schell, & Stottman, 1995; Office of 
Accountability [OEA], 1994; Raths, Katz, & Fanning, 1992; Raths & Fanning, 1993; Schack, 
1993) have shown that teachers in many contexts are moving toward more developmentally 
appropriate primary programs. However, the nature and quality of implementation is 
extremely varied. 

In many of the classrooms visited, children were engaged in hands-on activities, 
reading literature, writing compositions, and involved in conceptually-based mathematics 
activities. More than anything, children seem to be writing more than ever before and to be 
more enthusiastic about learning (Bridge, 1994, 1995; Coe et al., 1995; OEA, 1994; Raths & 
Fanning, 1993). Reportedly less textbook work, drill, seatwork, and rote memorization 
occurred in these classrooms (Aagaard, Coe, Moore, & Kannapel, 1994). 
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In a randomly-sampled, state-wide study of 86 teachers sponsored by the Kentucky 
Institute for Educational Research (Bridge, 1994), a four-point evaluation rubric (termed 
Innovation Component Configuration Map) was used to examine many aspects of the primary 
program, including such features as the physical arrangement of the classrooms, emotional 
climate, and instructional strategies. An "a" on the map described the most research-based, 
state-of-the-art practices, and a "d" described extremely traditional instruction. The evaluators 
determined that practices falling in the "a" and "b" categories, represented "acceptable" levels 
of implementation while "c" and "d" were unacceptable. In this study, at least half of all 
teachers were implementing "acceptable" practices in the areas of developmentally appropriate 
instruction, including purposeful movement, active engagement, student talk, student/teacher 
interaction, use of Kentucky’s learning goals, use of authentic problems, meaning-centered 
reading, meaning-centered writing, problem-solving mathematics, and discovery-science. The 
large majority of the "acceptable" levels were categorized as "b" rather than "a" showing 
room for improvement, even with many teachers who are successfully implementing the 
primary program. Fewer than half of the teachers observed in this study were found to be 
acceptable in the areas of learning centers, displaying student work, use of broad-based 
themes, and inquiry-based science. 

A follow-up to this study was conducted in 1995 (Bridge, 1995) to provide further 
data on implementation. The research design included 92 randomly selected classrooms in 24 
primary schools within Kentucky’s eight regional service center areas. The study examined 
general patterns of how teachers implemented the components of the primary program, 
whether and in what ways teachers implementing the program with high fidelity differed from 
those implementing the program with low fidelity, and what perceptions and attitudes teachers 
seemed to hold about support received and various aspects of the program. 

The findings of the 1995 study revealed little change in patterns of implementation 
from those of the 1994 study. Statistically significant decreases were found in teachers’ 
flexible arrangements of the physical environment and in providing for students’ continuous 
progress. Statistically significant increases were also found in teachers’ communication with 
special area teachers and involvement of parents in assessment of children’s progress (Bridge, 
1995). 



As in the 1994 study, half or more of the teachers used varied instructional materials, 
created supportive classrooms within which children could construct knowledge through social 
interaction, planned instruction within the framework of Kentucky’s Learning Goals and 
Academic Expectations, used qualitative methods of reporting student progress, and provided 
recommended instructional practices in reading, writing, and mathematics. Furthermore, as in 
the 1994 study, fewer than half of the teachers demonstrated recommended instructional 
practices in science and social studies and for integrating the arts, used varied grouping 
patterns to support continuous progress, incorporated learning centers, used varied 
instructional strategies, or used broad-based themes (Bridge, 1995). 

Other studies (Aagaard et al., 1994; OEA, 1994) augment the data found in the Bridge 
(1994, 1995) studies. In the Aagaard study, researchers observed 37 primary programs in ten 
schools in four districts for two days each (fall and spring). Researchers also analyzed 
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primary program action plans and interviewed teachers. They also found that teachers were 
best at implementing developmentally appropriate practices. By their description, those 
teachers would also fall primarily into category "b" on the configuration map. That is, while 
developmentally appropriate instruction was implemented to a further degree than other 
aspects of the primary program, teachers were still struggling with how best to integrate 
instruction based on Kentucky’s goals and provide meaningful language experiences. A few 
teachers (as in the Bridge, 1994 and 1995 studies) had not changed practices at all. 

The study by the Office of Education Accountability (1994) involved observations of 
75 schools in over 40 districts. They found that thematic instruction occurred in at least 
three-fourths of the classrooms; however, the mature of the themes, and whether they were 
related to the states’ goals and academic expectations, is not reported. Teachers in this study 
were also found to have a low level of implementation of learning centers and did little to 
accommodate for individuals (using a continuous progress model), thus confirming the studies 
by AEL and KIER (Kentucky Institute for Educational Research). 

Appalachia Educational Laboratory (AEL) revisited eight elementary schools in May, 
1994 that had been a part of their 1992-93 study (AEL, 1993) in order to determine progress 
in program implementation and questions for research during 1994-95. Although 
acknowledging that they were unable to draw definitive conclusions, the researchers were able 
to suggest preliminary findings (Coe et al., 1995). For instance, confirming the 1992-93 
findings, the critical attributes of developmentally appropriate practices and multi-age, multi- 
ability grouping seemed to drive teachers’ instruction and to reflect where they put most 
effort. The researchers noted a few problems, such as some reports of teachers returning to 
more traditional instruction and not integrating the curriculum to any great degree. However, 
in general, instruction seemed to be developmentally appropriate and, the researchers 
suggested, may be made easier with the statewide availability of the Kentucky Early Learning 
Profile (KELP) and its inclusion of performances for students to complete (Coe et al., 1995). 

The AEL study (Coe et al., 1995) also provided evidence that continuous progress 
remained the most difficult attribute for teachers to implement, again confirming their 
previous findings (AEL, 1993) and those of Bridge (1994, 1995) and OEA (1994). Also, 
more teachers in the AEL study reported concern about whether primary students were 
acquiring necessary basic skills. 

The study by McIntyre & Kyle (with colleagues Clyde and Hovda) (Kyle & McIntyre, 
1993, Gregory et al., 1995; Hovda et al., 1995; McIntyre et al., 1995) offers descriptions of 
practices in classrooms categorized mostly as "a" and explanations of how the teachers 
developed their programs. In this study, which carefully selected 10 outstanding teachers, 
classrooms were characterized by instruction from a constructivist perspective that was 
designed to meet the needs of all learners in the classrooms. These teachers differentiated 
instruction for the varied learners and worked^ either directly with children with special needs 
or extensively with special education personnel. They helped children make their learning 
explicit within meaningful activities. Their multi-age classrooms were social in nature and 
were characterized as authentic places for learning relevant material. The teachers 
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emphasized both processes and products. The teachers in this study have philosophical 
beliefs in line with a constructivist view of instruction, and they work hard to continue to . 
learn about best practices. Their participation in such experiences as a research project, 
writing workshop, community organizations, and graduate classes have helped them 
understand teaching and learning. Also, teaming and supportive administrators have been 
instrumental in their development as primary teachers. 

In the Raths and Fanning (1993) study, which was a follow-up evaluation of their 
1992 study of the implementation of primary programs, they also found that teachers’ whose 
philosophies were aligned with constructivist teaching were most successful in their 
implementation of the program. In a case stucty of two teachers with different theoretical 
orientations to literacy practices, Chmiel (1992) found that teachers of a traditional orientation 
were forced to rethink their practices in light of the mandate, but that certain policy 
constraints, not consistent with the reform, limited some of the changes they could make. 

Implementation of the multi-age, multi-ability component of the primary program was 
found to be varied in the studies. While AEL (1993) found this aspect to ha of the most 
commonly implemented attributes, it is clearly one of the most controversi; , u difficult for 
teachers (de Mesquita & Drake, 1994; Raths, Katz, & Fanning, 1992; Raths & Fanning, 

1993). In their 1994 revisit to schools, (Coe et al., 1995), AEL found that dual-age rather 
than multi-age grouping was preferred and had increased in practice. The researchers 
speculated that, "Dual-age grouping, because it results in the creation of distinct, age-level 
classrooms and groups, may be acting as a barrier to continuous progress" (Coe et al., 1995, 
p. 15). 



In some settings, teachers created multi-age classrooms for one hour a week (Aagaard 
et al., 1994; Bridge, 1994) while in others, teachers had created self-contained multi-age 
classrooms. Most classrooms did not combine more than two of the traditional grades. 
Arrangements in which children ages 5-9 with varied abilities are grouped in classes all day 
together are almost non-existent. (See McIntyre, 1995, for one exception). In the 1995 
statewide study, (Bridge, 1995), the findings indicated that 

Approximately half of the teachers were meeting the multi-age, multi-ability 
grouping requirement in ways that were originally recommended.. .that is, by 
grouping children in self-contained, dual-age and multi-age classrooms in 
which flexible grouping is used to meet the needs and interests of the 
children.. ..Approximately three-fourths of the children remain with the same 
teacher or the same classroom family for two or more years....Almost al! 
schools included special needs children in regular primary classrooms during 
all or most of the school day (Bridge, 1995, pp. xiii-xiv). 

Inclusion of five-year-olds is a controversial issue with many of the teachers (Bridge, 
1994; Raths & Fanning, 1993). However, some teachers who have included five-year-olds 
have reported exciting and dramatic results (McIntyre & Kyle, 1993; Raths & Fanning, 1993). 
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Many teachers still ability-group children for lessons (Aagaard et al M 1994) and many had 
negative attitudes about wide age spans in the classroom (Bridge, 1994). 

Bridge’s (1995) study found varied ways of including five- year-olds, with nearly half 
of the schools implementing programs that included five-year-olds with P2s. Almost one- 
fourth of the schools involved five-year-olds with two or more ages, and about 20% of the 
schools had established self-contained kindergarten classrooms. Of the schools that involved 
five-year-olds with other students, about half had full inclusion (or large blocks of time), and 
the other half involved them for more limited amounts of time or for special activities only. 



% 

A few teachers in some studies have said that having a wide age span in their 
classrooms has "pushed" them into more developmentally appropriate instruction (Bridge, 
1994). The ten teachers in the McIntyre & Kyle (1993) study found that having a wide 
variety of learners "forced" them to look at children’s development and to provide appropriate 
experiences for them. These teachers all have self-contained multi-age classrooms (several 
with three or more traditional grade levels). Most of these teachers no longer saw children as 
"kindergartners" or "third graders," but began to view them as individual children with 
strengths and needs. The teachers began to understand the concept of continuous progress 
more clearly and the link between continuous progress and developmentally appropriate 
instruction. These findings help explain why, in the Aagaard et al. (1994) study, they found 
the concept of continuous progress to be the least implemented. In that study, teachers were 
implementing research-based activities, but in ways that may not be best for the development 
of individual children. They had "activities" that looked appropriate, but there was little 
indication that teachers attended to the individual development of the children in their classes 
and differentiated instruction to meet their needs. Further, these findings help explain why, in 
the Bridge (1994) study, most teachers were categorized as "b" rather than "a" in the areas of 
developmentally appropriate instruction and continuous progress. Many teachers teach 
reading and mathematics to homogeneous groups and leave multi-age, multi-ability teaching 
for thematic lessons. 

Authentic assessment is also closely linked to developmentally-appropriate instruction 
and continuous progress. While more teachers are using strategies for authentic assessment 
(Bridge, 1994), many are struggling with how to use the notes and documents they collect on 
children (Aagaard et al., 1994), Currently, most teachers who do take anecdotal notes about 
children, conference and interview them, and collect documents do it for mechanical reasons, 
rather than to understand the learner or to make instructional decisions. The idea behind this 
mandate is to help teachers use the information from authentic assessment to inform best 
practices. However, this is difficult to do for most teachers (Aagaard et al., 1994; Bridge, 
1994; McIntyre & Kyle, 1993; Raths & Fanning, 1993). Even the "expert" teachers in the 
McIntyre & Kyle (1993; Gregory et al., 1995; Hovda et al., 1995; McIntyre et al., 1995) 
study found that using authentic assessment to improve practice is challenging, and that they 
need more professional development in this area. These same teachers have begun to use the 
Kentucky Early Learning Profile (KELP), an assessment tool that links assessment to 
instruction. 
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Clearly, teachers conversant on KELP and authentic assessment are the exception 
rather than the rule (OEA, 1994). Only about 800 teachers across the state have had training 
in the KELP. While it was envisioned that all teachers would implement the KELP during 
the 1994-95 school year, this is now viewed as too ambitious (OEA, 1994). However, use of 
this tool has been received favorably by the teachers who piloted it (Raths & Fanning, 1993) 
because it guided appropriate instruction and curriculum. Pilot teachers also found that use of 
the KELP helped them understand all the critical attributes of the primary program (Raths & 
Fanning, 1993). Certainly, with the addition of the "Learning Descriptors" which will 
accompany the KELP, teachers will be more able to understand children’s development and 
how to assess it (OEA, 1994). It may be that use of these tools will have a profound effect 
in the development in this area. Indeed, in Bridge’s (1995) study, teachers trained in the use 
of the KELP reported that the process was very time-consuming but worth the effort. They 
felt better able to monitor students’ progress and reported favorable reactions by parents. 

Factors influencing implementation. Many studies have reported that teachers feel a 
need for more time for planning, preparation, and refining their progress. Although some 
teachers said they now have more planning time than they used to, they still needed more 
(Bridge, 1994, 1995). Studies by Bridge (1995) and Coe et al. (1995) have confirmed this 
view held by teachers. Many studies reported that teachers wanted and needed school time 
on a daily and weekly basis (AEL, 1993; Craig, in progress; Kyle & McIntyre, 1993; Raths, 
Katz, & Fanning, 1992; Raths & Fanning, 1993). Others wanted the timeline of the mandate 
to slow (de Mesquita & Drake, 1994; Raths, Katz, & Fanning, 1993) because they felt the 
fast pace of the reform was unrealistic and unattainable. Teachers have spent evenings and 
weekends at school trying to accomplish all that is expected of them. Raths, Katz, & Fanning 
(1992) described teachers who held bitter confrontations with spouses who thought the 
teachers should worry as much about their own children as they do about the children in their 
classrooms. Teachers reported that they were working fewer hours now than during the first 
year of implementation but more than before KERA (Raths & Fanning, 1993). 

Importantly, many studies have reported that when given support, teachers were able 
to make changes more easily (AEL, 1993; Bridge, 1993; Chmiel, 1992; Kyle & McIntyre, 
1993; Raths, Katz, & Fanning, 1992; Raths & Fanning, 1993). AEL suggested that the 
principal is key to successful implementation, citing how supportive instructional leaders 
encouraged and provided resources and time for their teachers. Principals who have helped 
re-schedule the school day in order for teachers to have common planning time have been 
instrumental in helping teachers make changes (Kyle & McIntyre, 1993). A supportive 
principal is particularly helpful in defending the program in public meetings (Raths & 

Fanning, 1993). Raths, Katz, and Fanning (1992) suggested that support can be obtained 
through a sympathetic principal, additional resources, enthusiastic colleagues, and/or a trusting 
community. 

One study which examined the conditions affecting successful implementation of the 
program (Gooden, 1995) supports evidence cited above. Gooden conducted eight case studies 
of teachers with a "high fidelity" to recommended practices. She found that these teachers a) 
were highly committed to the program, b) perceived the program to be compatible with them 
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as teachers, c) believed in continuous progress, d) had teacher efficacy, and e) held 
empathetic understandings of children and other teachers. Thus, teacher characteristics may 
be a factor in the successful implementation of the program. Anecdotal evidence suggests 
that the teachers in the McIntyre & Kyle study (1993; Gregory et al., 1995; Hovda et al., 

1995; McIntyre et al, 1995) have similar characteristics. 

The factors which affected the successful implementation of the programs of the 
teachers in the Gooden (1995) study also augment previous studies. She found that across the 
schools in which these successful teachers worked, supportive principals and collaborative 
relationships existed among the faculty. Gooden concluded that teachers are instrumental to 
the success, but that the principal is key as well, and that teachers who are professional, 
committed, cooperative, and child-centered can be successful. 



What have been the effects of the program on stakeholders? 

A common finding among the studies was the increase in amount of collaboration 
among teachers and other professionals, as well as parents. Some studies report that the 
professional teamwork mandate is enthusiastically endorsed by the teachers (Kyle & 
McIntyre, 1993) and that increased efforts are being made to involve and collaborate with 
parents (AEL, 1993; Bridge, 1993, 1994, 1995; Raths, Katz, & Fanning, 1993). Parents 
perceived schools as being "closer knit" (Raths, Katz, & Fanning, 1992, p. 10). Although 
parents are more involved in the schools than they had been, they are still involved in fairly 
traditional ways (Bridge, 1993, 1994). 



Primary teachers report more joint planning, communication, and collaboration among 
themselves (Aagaard et al., 1994; Bridge, 1994; McIntyre & Kyle, 1993) and, to some extent, 
with special education teachers (Aagaard et al., 1994; McIntyre & Kyle, 1993). However, 
many teachers report that they still do not have joint planning time with teammates (Bridge, 
1994, 1995) and that their team interactions are occasional (Bridge, 1994). In the McIntyre & 
Kyle (1993) study, the teachers have reported that teaming has been an essential part of their 
development as primary teachers. In three of the four schools, teachers actually teach 
together in one room the same group of children, and planning and collaboration occurs daily. 
In the fourth setting, the teachers are committed to team planning weekly where they decide 
many of their broad-based themes and share instructional resources and ideas, problems, and 
solutions. Some of the teachers in this study had participated in the research project and had 
been involved in professional writing together for over three years. They reported that having 
a common goal or mission has been factor in the development of their highly "connected" 
teams who make all major decisions together. In another recent study (Tackett, 1993), some 
primary teachers’ instruction was found to be influenced by the performance assessments 
conducted in the fourth grade. However, in the same study, many primary teachers expressed 
ignorance about the nature of fourth grade assessment. Johnson and Bruno (1993) report that 
special educators say they need more money, time, and concrete examples of how to 
collaborate successfully with general educators to meet the needs of children with disabilities. 
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Little research exists to date, on the KERA Primary Program’s effect on students. 
However, de Mesquita & Drake (1993), who interviewed and surveyed children, reported that 
students did not appear to be at all stressed about the changes, and Craig (1993) reported an 
apparent diminished number of discipline problems in the primary programs. Hughes (in 
progress) reported that with the use of authentic assessments, children’s cultural backgrounds 
were more respected. One case study of a primary teacher with extensive experience with 
developmentally appropriate practices suggested that the children were more "empowered," 
which allowed them to become more actively involved in their own education (McGee, 1994). 
Teachers in Raths & Fanning’s (1993) study reported that students are writing more, are 
better informed about the world, have more positive attitudes toward school, and have higher 
attendance rates than in preKERA classrooms^ Another study on the relationship between the 
primary program and school attendance is also underway (Oakes & Mann, in progress). 

Several studies reported differing effects for exceptional children. McIntyre (1995) 
found overwhelmingly positive academic and emotional effects for three children with 
learning disabilities placed in the primary program classroom all day with both regular and 
special education teachers. Schack (1993), in her case study of six exceptional learners in 
one school, found that the primary program appeared to have positive academic, social, and 
emotional effects for younger, brighter children and older, less able students. However, there 
was concern that the primary program was not meeting the needs of the older and brighter 
children in the classroom. Evans (1994) in an interview study of 36 primary teachers about 
meeting the needs of gifted primary students concluded that "...in actual practice few primary 
students are being identified and provided services in or out of the primary classroom" (p. 

36). No study has systematically examined children’s academic performance on a large scale. 

Collaboration with special education teachers and the effects of it have been studied 
by researchers at Northern Kentucky University. Through observation and interview data 
collected in Kentucky and British Columbia classrooms, Bruno studied the Primary Program’s 
effects on services for students with disabilities. Specifically targeted for study were such 
factors as the rate of referrals, the outcomes accomplished by students with disabilities, and 
indicators of successful integration. Recommendations for teacher education were offered. 

Bruno offered the following conclusions from her study: (1) teachers’ successful 
inclusion of students with disabilities is influenced by such factors as administrative support, 
prior education and experience, personal values, and sociocultural factors; (2) guidelines from 
the teachers’ union in British Columbia regarding the acceptable number of students of 
particular ages or with differing degrees of handicap severity in a multiage grouping had a 
greater influence than practices recommended by the Ministry of Education; (3) the more 
extensive professional development opportunities in British Columbia have enabled teachers 
there "to have a more thorough application of the concepts of developmentally appropriate 
practices and continuous progress in their classrooms" (Bruno & Johnson, 1994, p. 2); (4) the 
tradition of job sharing in British Columbia has created the need for teachers to interact and 
collaborate. Kentucky teachers are less experienced in professional teamwork; and (5) the 
lack of accountability measures in British Columbia "seems to have an influence on the 
positive atmosphere of meeting individual needs and awareness of student readiness 
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levels.. .(Bruno, 1994, p. 3)." A few of the Kentucky teachers in the study expressed concern 
that their school’s KIRIS test scores would be affected by the inclusion of special needs 
students. 

The most recent study of the effects of KERA on children’s achievement (Hughes & 
Craig, 1994) compared children’s achievement on the Kentucky Instructional Results 
Information System (KIRIS) performance assessments to levels of implementation of the 
primary program. These KIRIS assessments include open-response questions in reading, 
mathematics, science, and social studies. Researchers asked teachers in 1 1 schools to self- 
report their implementation levels of the primary program, and outside evaluators ranked each 
teacher’s implementation level on each of the pritical attributes. Discrepancies among 
researchers about each teacher’s evaluation were resolved by consensus. Each school 
received an implementation level score. These data were then correlated with children’s 
performance on the KIRIS performance assessments. While some schools showed remarkable 
achievement on the assessments across the second and third years of implementation, there 
was no general pattern that could be associated with the scores on the level of implementation 
of the primary program. While no clear relationships exist from this research, this avenue of 
study represents the beginning of needed studies on the effects of the primary program on 
children’s learning. 



What are the attitudes and perceptions of stakeholders toward the program? 

Prior to the mandated implementation year, Hovda (1992) examined teachers’ and 
principals’ understandings and attitudes about implementing the KERA Primary Program 
mandate. This study surveyed teachers and principals across the state (representing all 
geographic areas) on beliefs about how children learn and attitudes toward specific aspects of 
the mandate. In analyzing the 6,929 responses received, Hovda (1992) found that despite 
some lack of understanding of the reform, teachers and principals generally held very positive 
attitudes toward the reform, believed they could implement it appropriately, and viewed the 
change as likely to be good for children. Further, most teachers perceived themselves as 
having a more positive attitude than their peers, suggesting that despite some opposition to 
the primary program, more teachers felt positively toward the mandate than did not. 

In a later, more in-depth study (using surveys, interviews, and observations) 
de Mesquita and Townley (1993) also reported that teachers were generally positive about the 
reform and confident about implementing it. However, the researchers also suggested that 
teachers in their study wanted to slow the process, that they felt too much pressure in 
implementing all aspects of the reform simultaneously. Researchers de Mesquita and Drake 
(1994) found that most teachers believed that the primary program would be effective in 
improving the overall performance of students, and that more than half said they preferred 
teaching in the reformed primary program rather than the traditional arrangement. However, 
83% of the teachers also believed successful implementation would require additional skill 
and technical retraining. The areas the teachers felt most unsure about, in terms of their 
abilities to implement the reform, included: conducting authentic assessments; monitoring 
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continuous progress; developing qualitative reporting methods; and arranging multi-age, multi- 
ability groupings. A study by the Office of Education Accountability (OEA, 1994) confirmed 
many of the same findings concerning teachers’ attitudes. They also report that teachers were 
positive about the most recent 1994 decision that classrooms can be dual-age instead of multi- 
age. Very traditional teachers were also found to be concerned that they were going to be 
viewed as "less excellent" than they had been viewed prior to the mandate (Raths, Katz, & 
Fanning, 1992). Further, de Mesquita and Drake (1994) found that those with less teaching 
experience tended to be more knowledgeable about KERA than those with more, and that 
those with the least and the most years of teaching experience welcomed the reform more 
than those considered to be in the mid-stage of their professional careers. 

A study by Addington and Hinton (1993) surveyed 37 teachers from one school 
district. Their findings indicated that most teachers felt capable of implementing 
developmentally appropriate practices and perceived that the program created an enjoyable 
atmosphere within which both teachers and students were excited about learning. 

However, two other studies found that principals and teachers report more conflict in 
their jobs than ever before. While Johnson and Bruno (1993) found that conflict resulted 
from the lack of collaborative relationships among special and general educators, de Mesquita 
and Ballard (1993) found that, although there was more conflict in the schools, most teachers 
felt confident about managing the conflict. Many primary teachers viewed the conflict as a 
necessary part of change. 

Pittman and Hinton (1993) focused on children’s attitudes about reform. They 
administered questionnaires to 53 primary and 47 fourth-grade students and found that the 
students were accepting of the changes and adapting to thematic teaching and cooperative 
learning. Primary students more than the older students reported a preference for working in 
cooperative groups. 

Cantrell (1994) reports that many studies, a^well as anecdotal evidence, show that 
parents’ attitudes and perceptions of KERA vary wi3b(y. At a forum held at a state 
university, parents voiced concerns that some of the goals of KERA (for example "Become a 
self-sufficient individual") were not measurable and therefore should not be taught. Others 
were concerned about the cost of reform, the teaching of values, lack of testing and 
accountability, and lack of focus on the basics. 

A recent study sponsored by the Kentucky Institute for Educational Research (KIER) 
and conducted by Tom Wilkerson and Associates, Inc. involved a statewide telephone survey 
of several constituencies-parents, teachers, principals-to elicit opinions about KERA 
(Wilkerson, 1995). More than half of all participants were in agreement with seven of the 
eight posed belief statements concerning KERA, such as, "All children can learn at high 
rates" and "We should set high standards of achievement for all children." However, one 
statement with which most respondents disagreed was, "In the primary schools, students 
should not be labeled as belonging to a specific grade level." The response to this statement 
reflects other studies which report the lack of understanding by teachers and others on the 
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concept of continuous progress. Further, many of the attitude statements reflect more 
negatively about KERA than they did a year ago. This may show some division among 
educators who are knowledgeable about KERA and practicing appropriate instruction as 
compared to those who may be less knowledgeable. 

Indeed, primary teachers who are more knowledgeable about the primary program than 
their peers v-ho teach at higher grade levels are more apt to perceive that the primary program 
is working well. When the views of primary teachers were compared to those of 
intermediate, middle school, and high school teachers, Wilkerson (1995) found that 61% of 
primary teachers believed that the primary program was working well; whereas, only 34% of 
the intermediate teachers and 29% of the middle school and high school teachers judged the 
primary program to be working well. Furthermore, Wilkerson found that educators in reward 
schools were more likely to make statements such as, "The primary program is working well" 
and, "All children can learn and most at a high level" than teachers and administrators in non- 
reward schools. Again, this finding supports the notion that better understanding of the 
primary program and the goals of KERA is corollary to more positive attitudes. 



What research is in progress in this area of KERA? 

Several studies on the primary program are in progress or anticipated. Those known 

at this time include: 

1 . The Kentucky Institute for Education Research conducted a survey of superintendents 
and local school board members in Summer 1995. Some of these questions address 
the primary program. Results are forthcoming. 

2. Roberts at Western Kentucky University anticipates reporting in Spring 1996 the 
results of a three-year study of primary gifted education. The data include narrative 
and questionnaire responses from teachers, classroom observation data, and student 
data. 

3. The Kentucky Institute for Education Research plans to continue funding in 1996 for 
its annual study of the implementation of the primary program which has been 
coordinated by Bridge at the Institute on Education Reform at the University of 
Kentucky. 

4. Coe and colleagues at the Appalachia Educational Laboratory conducted further 
research on the primary program in 1994-95. Findings will be reported in their annual 
report and in a book being developed for publication. 

5. McIntyre, Hovda, and Kyle at the University of Louisville are initiating a longitudinal 
study of primary students’ academic and social development. 
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Which questions or issues need further research? 

Due to studies which show growing negative attitudes t-bout KERA primary programs, 
research is needed on why educators feel negatively and the conditions under which they 
work. Correlations of attitude, knowledge, and practice would also shed light on why some 
feel so negative about the reform. Are these teachers working under similar conditions? Do 
they really understand the critical attributes? Are they regionally located? Do they get the 
professional development they need and want? These data coul(TT!av£4inplications for 
professional development and, ultimately, for the implementation of the primary program as it 
was intended. 

* 

Since the data indicate positive responses from primary program teachers who are high 
implementers and from those who teach in reward schools, studies on the factors which help 
explain successful implementation and positive attitudes could also be enlightening and 
useful. 



As noted in previous reviews, research conducted to date has provided information that 
can be useful for such constituencies as teachers, administrators, teacher educators, 
professional development directors, parents, and policy makers. However, as stated above, 
further research on KERA Primary Programs is critically needed in order to make informed 
decisions about how to revise and refine the program, how to enhance the likelihood c" 
student success, and how to support teachers’ professional development. Research 
possibilities exist for university faculties. Department of Education staff, district-level 
personnel, other interest groups, and for classroom teachers themselves who may be interested 
in action research (Williams [in progress]). Collaborative projects involving researchers 
representing various constituencies could be especially useful. 

In summary, teachers appear to be farther along in the development of their primary 
programs. Many teachers are "pro-KERA" due to their understandings of it (Raths & 

Fanning, 1994) and have made good faith efforts to implement it (Bridge, 1994, 1995). But 
most are still implementing the "nuts and bolts" of the program (Bridge, 1994, 1995). 

Teachers are currently focusing on the components of the program rather than the whole 
picture. This may be due to time constraints (Aagaard et. al, 1994; Raths & Fanning, 1993) 
or to lack of the knowledge, skills, and beliefs necessary to implement the program (Bridge, 
1994, 1995). With time for planning, continued professional development, and financial 
backing, most teachers will be able to successfully implement this program. 

Studies on the implementation of the various attributes of Kentucky primary programs 
need to continue in order to assess how this implementation is modified over time and why, 
what barriers appear to constrain implementation of specific aspects of the program, and what 
factors appear to facilitate successful implementation. Long-term, in-depth studies of a small 
number of programs as well as more short-term studies that involve many programs are 
equally needed. 
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And, finally, now that many programs have been implemented for three to four years, 

future research studies need to focus more explicitly on the effects on students. Previous . 

reviews offered several potential research questions on this topic and are repeated here: 

1. How do children grow socially, emotionally, physically, and aesthetically in primary 
programs? 

2. How has the use of the Kentucky Early Learning Profile affected instruction and what 
children leam? 

3. Are there groups of children (i.e., the learning disabled, gifted, culturally different, 
males, females) who seem to benefit most from primary programs? Least? Why? 

4. What are students’ perceptions of the primary program? 

5. What effect does having the same teacher or teachers for two or more years have on 
children’s progress? 

6. What impact does parent involvement have on student learning? 

7. Do students in dual-age or multi-age settings perform better than students in traditional 
graded settings? 



8. What is the effect of a holistic instructional focus? Do children leam enough of the 
"basics?" If so, how? What is the relationship of instruction to learning? 



All of these questions need exploration to provide greater understanding of the impact 
of the KERA Primary Program on students and teachers and to determine the strengths and 
weaknesses of the program so that further refinements may be made. 
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HIGH SCHOOL RESTRUCTURING 



John Fischetti and Allan Dittmer 
University of Louisville 



What does the law or regulation mandate? 

The Kentucky Education Reform Act of 1990 (KERA) stipulates that schools will be 
measured, in part, on their graduation rates and on the proportion of students who make a 
successful transition to work, post-secondary education or the military. In addition, KERA 
mandates in Kentucky Revised Statute 156.160 that "prior to the beginning of the 1994-1995 
school year, the State Board for Elementary and Secondary Education shall review graduation 
requirements in light of the expected outcomes for students and schools set forth in Kentucky 
Revised Statute 158.6451 (Learner Outcomes, now the Academic Expectations)." 

Prior to KERA, Kentucky high schools were involved in a variety of restructuring 
activities. For example, a number of schools participated in national projects or programs that 
were aimed at school change. These included the Coalition of Essential Schools, the 
Effective Schools Project, the National Writing Project, and the Foxfire teacher network. 

Other restructuring initiatives included Tech Prep, professional standards development in 
various content areas, isolated occurrences of team teaching and interdisciplinary curricula, 
and a variety of school/university partnerships. 

In. July 1992, the State Board for Elementary and Secondary Education asked the 
Kentucky Department of Education to appoint and manage a High School Restructuring Task 
Force composed of citizens and stakeholders. In June, 1993, the Task Force presented its 
final report to the State Board, proposing the establishment of pilot sites. In November 1993, 
the State Board approved the Commissioners recommendation and the application procedures 
for funding a network of 27 "Developmental sites” and 40 "Mini-Grant sites for 
restructuring." The Developmental Sites each received $7,900 to support piloting of the new 
graduation requirements recommended by the Task Force. Mini-Grant sites received $2,000 
to implement one aspect of restructuring. Some schools have chosen to implement the 
recommendations on their own, independent of formal funding. This brings the total number 
of schools in the state piloting high school restructuring to approximately 100 or about 1/3 of 
the total number of high schools. In 1996, after a two-year process of piloting these new 
requirements, KDE will present recommendations to the State Board regarding formal changes 
in high school graduation requirements. We are currently in the second year of the pilot 
program. 



Performance-Based Graduation Requirements: 

In order to develop performance-based graduation requirements that honor the six 
KERA Learning Goals and 57 Academic Expectations, in its report the Task Force proposed a 
locally determined, phased-in process. The development of new performance-based 
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graduation requirements necessitates a re-evaluation of all current curriculum, including 
precollege and vocational/technical curriculum. The Task Force agreed that schools should, 
for the time being, continue to use Carnegie Units (i.e., the unit used to define one year of 
study in a high school subject-which must be at least 120 sixty minute hours), however, 
they also agreed that schools may expand evidence of student learning by piloting new 
performance-based credits over an extended period. The Task Force proposed a timeline for 
the development and implementation of the new requirements (see Task Force on High 
School Restructuring, 1993, p. 21). 

Proposed Core Components for High School Graduation: 

\ 

Following is a brief summary of the proposed required core components for high 
school graduation taken from the Task Force report (see Task Force on High School 
Restructuring, 1993, pp. 22-24 for complete descriptions). These core components are being 
piloted by the Developmental Sites. 

1. Individual Graduation Plan 

Prior to entering high school, with the guidance of parents and educators, each student 
shall develop an Individual Graduation Plan that documents an academic program of 
study for achieving the six KERA Learning Goals and demonstration of the Academic 
Expectations. As part of the plan, students shall indicate a program of study that 
enables them to complete high school and be eligible for each of the following: 
college, vocational/technical school, the workforce (or home as a workplace), and the 
military or community service. The plan will include specific academic courses and 
projected school-sponsored or approved activities. 

2. Integrated Academic Portfolio (now Profile) 

The student shall maintain a required Integrated Academic Profile for the years he/she 
is enrolled in high school. The student would assemble a single portfolio from all 
courses and experiences throughout high school. The Academic Portfolio includes a 
transcript; a resume; appropriate test data (such as ACT, SAT, etc.); statewide 
assessment results; recommendations from educators and employers; certificates and 
awards; print and non-print examples of performance, demonstrations and/or 
exhibitions; documentation of satisfactory participation in school sponsored and 
approved activities and the culminating project. 

3. Student-Initiated Culminating Project and Panel Presentation 

During the review process of the Individual Graduation Plan, prior to the anticipated 
final year of high school, the student shall design a significant culminating project. 

The Culminating Project will include a major written component supported by 
appropriate documentation, references and research; and an oral or visual performance, 
demonstration, exhibition or presentation to be presented to a panel. 
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4. Required School Sponsored and Approved Activities 

The student shall actively participate in at least one school sponsored or approved . 
activity during each year he/she is enrolled in high school. In addition, during the 
years he/she is enrolled in high school, the student shall actively participate in any two 
of the following activities: (a) service learning, meaningful activity that benefits the 
community; (b) school service, a meaningful activity that benefits the school, school 
personnel or other learners; (c) work-based learning, a work program, internship or 
simulation with predetermined learning goals, and; (d) student-initiated enrichment-a 
personally enriching activity or experience that complements the student's graduation 
plan. 

5. Exit Review 

The components required for high school graduation will be verified by the school 
official so designated. 

6. Exceptions 

The school council/district will develop policies and procedures to individualize the 
graduation requirements for students who have special needs or with extenuating 
circumstances. 

Fundamentals of High School Restructuring: 

In addition to the core components, the KDE team working on implementing 
restructuring subsequent to the Task Force, developed 1 1 “fundamentals” of high school 
restructuring. The process components are "pieces" of the restructuring puzzle that assist in 
clarifying for schools some of the smaller steps that might be taken on the complex road to 
systemic change. The "Mini-Grant" sites were each funded to pilot one of these components: 

• core curriculum 

• curriculum redesign 

• student engagement 

• performance standards and accountability 

• professional development 

• structure and organization of schools 

• technology 

• alternative uses of school time 

• school-wide engagement 

• successful transition to adult life 

• community participation 

KDE will report to the State Board for Elementary and Secondary Education at the 
completion of the 1996 school year as to the implementation of both the larger and smaller 
initiatives. 
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Review of Research to date. 



To what extent has the program been implemented as intended by KERA? 

Prior to 1994, there was no systematic research taking place to examine high school 
restructuring in Kentucky. Since that time the following efforts have begun: 

1. The Kentucky Institute for Education Research (KIER) funded a study of 33 Kentucky 
High Schools, including 12 which are developmental sites, 1 1 mini-grant sites and 10 
other schools involved in restructuring. A final report of the study is available from 
KIER. Researchers used an Innovation Configuration Map developed by KIER and 
formal interviews with multiple constituents as the primary research methods 
(Fischetti, 1995). The findings are as follows: 

What are the overall implementation factors crucial to the level of engagement in high 
school restructuring? It was found that the following eight factors were associated 
with meaningful engagement in restructuring: 

• Visionary and supportive principal. 

• Teachers fully involved in decision-making. 

• Counselors leading the change. 

• Students actively involved in change. 

• Funding reallocated to support teachers. 

• Standards established for new requirements. 

• Parents and community supportive of change. 

• Time used in new ways. 

In the next few years, what will likely be the significant high school restructuring 
initiatives? Future efforts at restructuring are likely to be characterized by the 
following: 

• Shared-decision making with all constituents fully involved. 

• Increasing use of teacher teams. 

• More flexible use of time by those teams. 

• Student learning assessed by multiple means, including performance-oriented 
graduation requirements. 

• Increasing links of curriculum to workplace and other post high school 
environments. 

• Increasing adoption of year-round school schedules. 

• Increasing use of multiple forms of data to determine school success. 

Recommendations of the KIER study. 

a. All high schools should design and implement formal plans to evaluate the progress 
of restructuring and the impact on students (including special populations, 
educators, the school and the community). 
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b. The recruitment, preparation and support for the next generation of school leaders 
should become a high priority of the Education Professional Standards Board, the 
Kentucky Department of Education and Kentucky institutions of higher education. 

c. The Department of Education should modify the state’s school assessment and 
accountability system to minimize incentives for short-term strategies to achieve 
high KIRIS scores at the expense of efforts to achieve long-term restructuring 
goals. 

d. High schools should develop specific strategies to gather and use students’ input 
and ideas in the planning, development, implementation and assessment of 
restructuring initiatives. 

e. High schools should work more directly with KDE staff to explore waiver options 
for programs of study available to them that facilitate their restructuring efforts. 

f. Local schools, the Department of Education, institutions of higher education and the 
media should facilitate public dialogue and discussion about the role, purpose and 
organization of high schools in the future. 

g. The Kentucky Department of Education should create a division or administrative 
unit to manage the multiple issues affecting schools. 

h. High schools with exemplary restructuring components should be identified, and 
descriptions of the successful innovations widely disseminated to high schools 
throughout the Commonwealth. 

i. Successful strategies for involving parents, business partners and community 
patrons need to be developed and/or disseminated to high schools involved in 
restructuring. 

j. The KDE and higher education institutions need to develop a greater capacity for 
providing professional development with respect to specific high school 
restructuring initiatives that have been identified as high priorities. 

k. District offices should explore and develop supportive roles for central office staff 
relative to high school restructuring. 

l. All high schools should develop formal linkages with feeder middle schools to 
facilitate a continuous and supportive curriculum, middle schools through high 
school, for all students. 

2. Each school selected as a developmental or mini-grant site is required to have an 

evaluation component, including a portfolio that includes qualitative and quantitative 
methods. The Kentucky Department of Education (KDE) is undertaking a self-study 
of selected sites in order to make recommendations to the State Board of Elementary 
and Secondary Education. An interim report (King, 1995) was submitted to the State 
Board by KDE, with a comprehensive report due upon completion of the two-year 
pilot. 

According to the interim report, all 27 developmental sites were visited in May and 
June 1995 by KDE personnel (King, 1995). Surveys and interviews were conducted 
using similar instrumentation as was reported by Fischetti (1995) in the study 
sponsored by the Kentucky Institute for Education Research (KIER). In the latter 
study, "Configuration Maps" were completed for schools, indicating how much 





progress had been made in implementing restructuring initiatives. The goal of the data 
collection reported by King (1995) was to determine the level of implementation of . 
the five core components of restructuring at the 27 Developmental Sites. 

None of the Developmental Sites achieved full implementation of any of the core 
components. However, various levels of progress were evident on all of them (King, 
1995). A seven-step scale of implenr station was employed, with the levels named as 
follows (from 1 meaning the most implemented, to 7 meaning the least implemented) : 
(1) Refinement, (2) Implementation, (3) Early Implementation, (4) Advanced Planning, 
(5) Planning, (6) Early Planning, and (7) Not In Place. 

For two of the five core components, more than 50% of the developmental sites were 
at the Implementation or the Early Implementation phases. For the core components, 
exact percentages of the sites at these two phases were: Individual Graduation Plan, 
69%; Integrated Academic Profile, 25%; Required School Sponsored and Approved 
Activities, 58%; Culminating Project and Panel Presentation, 35%; and Exit Review, 
22 %. 

3. The UK/UofL Joint Center for the Study of Educational Policy funded a grant to 
assess the research related to the implementation of KERA at the high school level 
(The Status of the Kentucky Secondary School: Simpson, Sandidge, Dittmer, Fischetti, 
& Stroble, 1995). The study summarizes the characteristics of Kentucky high schools 
and summarizes sources of information about restructuring efforts. Information crucial 
to understanding restructuring includes data on enrollments, expenditures, governance, 
student retention and transition data, school-to-work figures, community involvement, 
and delivery of instruction. The report also provides recommendations to the various 
education community constituents regarding restructuring. 

4. Graduate students at several of the 'state’ s colleges and universities are immersed in 
studying varying aspects of restructuring. Working with their advisors, graduate 
students are currently involved in studies including: 

• the impact of restructuring on special populations 

• the effects of high school restructuring in general and in some parts such as 

SBDM, portfolios, leadership, assessment, professional development, etc. 

• the politics of reform 

• the varied use of time 

There is no coordinated effort at the state universities to link the activity of graduate 
students who are studying various aspects of KERA and high school restructuring. 
Thus, the information on what research is being conducted is vague and anecdotal. 
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What have been the f ff ects of the program on stakeholders? 



As the previous review indicates, high school restructuring is a complex connection of 
all aspects of change in high schools. The long-term effects of restructuring can not yet be 
documented. In the short run, high school restructuring has been initiated by most schools by 
changing the daily master schedule. There are no data to determine the impact of specific or 
overall changes on student learning. And, as the KIER study reports, little evidence is being 
collected about change in schools to determine the effects of implementation of any initiative 
other than by comparing KIRIS results. 



* 

What are the attitudes and perceptions of stakeholders toward the program? 

There has been minimal research regarding attitudes and perceptions about high school 
restructuring. The KIER study suggests that attitudes and perceptions at individual schools 
are a function of the vision and personality of the building principal. The study found 
positive consensus toward change in about one-half of the schools included in the research. 

In rare cases were student perceptions gathered in any way as to inform the change process. 
Restructuring is such a multi-dimensional process that attitudes about the larger process of 
change are intertwined with individual personalities, school cultures, community expectations, 
and the specifics of each initiative. 



What research is in progress in this area of KERA? 

Many of the studies mentioned in the previous section on Review of Current Research 
have continued in 1995. For example, the pilot "developmental sites" and "mini-grant sites" 
are continuing. After reviewing data from these sites, the KDE recommendations for changes 
in high school graduation requirements are set to be issued in 1996. 

Also continuing are the studies of restructuring by graduate students at the state's 
universities. 



Which questions or issues need further research? 



1. Follow-up, longitudinal studies should be conducted in as many schools as possible 
that are involved in restructuring efforts. Included should be studies of the impact of 
restructuring on student performance. Studies should be conducted to continue to 
track the progress of high school restructuring. 

2. Case studies and vignettes of particularly successful high schools should be developed 
to examine and determine those components contributing to the level of restructuring 
and the long-term impact on student performance. 
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3. On-g6ing data should be formally collected from practitioners regarding restructuring 
initiatives. 

4. A review of high school restructuring across the United States should be conducted, 
comparing and contrasting Kentucky’s efforts. 

5. Research studying the impact of school/university collaboration and the impact on high 
school restructuring should be initiated. “ 

6. The state universities should form a “clearing house” on KERA and restructuring 

related research that is being conducted by faculty and graduate students. This listing 
should be updated regularly. « 




138 



182 



REFERENCES 



Fischetti, J. (1995). Assessing the impact of high school restructuring in Kentucky . 

Frankfort, KY: The Kentucky Institute for Education research. 

King, D. E. (1995). Developmental site interim report . Frankfort, KY: Kentucky 
Department of Education. 

Simpson, K., Dittmer, A., Fischetti, J., Lowe, M., Sandidge, R. & Stroble, E. (1995). The 

status of the Kentucky secondary school . Final report to the UKAJofL Joint Center for 
the Study of Educational Policy, Lexington, KY. 

Task Force on High School Restructuring (1993). Task force on high school restructuring 
final report . Frankfort, KY : Kentucky Department of Education. 




139 



183 



EXTENDED SCHOOL SERVICES 



Jeffery C. Drake 
University of Kentucky 



What does the regulation mandate? 

The Kentucky Education Reform Act (KERA) has required restructuring of certain 
aspects of schools in order to create a more supportive learning environment for all students. 

A key ingredient of KERA is the mandating of Extended School Services (ESS) programs to 
provide continuing education programs for students who need more time to be successful. 

KRS 158.070 states, "Schools shall provide continuing education for those students who are 
determined to need additional time to achieve the outcomes defined in KRS 158.6451, and 
schools shall not be limited to the minimum school term in providing this education. 
Continuing education time may include extended days, extended weeks, or extended years" 
(p-217). 

As required by state law, the Kentucky Board of Education approved administrative 
regulations that give important guidelines for implementing the ESS program. The 
administrative regulation, 704 KAR 3:390: 

1. Establishes criteria for the allotment of grants to local school districts; 

2. Defines additional instruction time to mean extended school services which occur 
before or after school, in the evenings, on weekends, during the summer and during 
extended school years; 

3. Provides a selection and referral process for students; 

4. Defines the instructional program and support services for ESS; 

5. Provides an appeal process for parents; 

6. Provides a process for accountability and evaluation; 

7. Provides competitive grants for innovation. 

In 1992, the state legislature amended KRS 158.070 with the passage of House Bill 
596 which allowed district school boards to adopt a policy to mandate attendance to ESS. 

The policy must specifically describe the students to whom it applies. District policy may not 
require students to attend ESS for punitive purposes or to "make-up time." Up to this time, 
only 8-10 districts have adopted a mandatory ESS attendance policy and most have been 
aimed at high school students. 

In April 1994, the Franklin County Circuit Court ruled that private school children in 
Campbell County and Fayette County could not be denied access to ESS-provided summer 
school based solely on their status as private school children. School districts across the state 
were informed of this ruling along with a recommendation to abide by the ruling unless 
otherwise legally advised. 
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Review of Research to date. 

To what extent has the program been implemented as intended bv KERA? 

School districts were required to implement ap Extended School Services (ESS) 
program in the first year of the Kentucky Education Reform Act. By November 1990, all 
176 school districts submitted applications to receive ESS grants for implementing an ESS 
program. In the first year of the program which included a majority of after school programs 
and summer programs, 92,228 students from primary through the twelfth grade were served. 
This number increased in 1991-92 regular and summer sessions to 120,006 students. These 
numbers have steadily increased with 135,557 students served in 1992-93, 149,222 served in 
1993-94 and 1993-94 figures project over 150,000 students served in regular and summer 
programs. 

Demographic data 
indicate that 54% of all 
students served were male 
while 46% were female. 

Ethnic status data 
demonstrate that 88% of all 
students were Caucasian, 
while 12% were minority 
students. This number 
represents a slight decrease 
in the number of minority 
students being served. The 
number of students being 
transported by ESS 
programs (42%) is 
significantly higher than 
prior years and may 
indicate a trend towards increased 

The number of students served has been somewhat evenly distributed across all grade 
levels during the first five years. During the 1994-1995 school year 54 percent of the ESS 
population was served in grades K-6, 26 percent in grades 7-9 and 20 percent in grades 10- 
12. Overall, there appears to have been a slight increase in the number of students served at 
the secondary school level. 



Grade Level of ESS Students 




service provision. 
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After school programs serve 90 percent of all students and continue to be the 
predominant service delivery model used throughout the state largely due to transportation . 
and school, student, and parent preferences. Before school, Saturday and evening programs 
have often been utilized at the secondary level where student employment and extracurricular 
activities (i.e., sports, band, and clubs) interfered with after school attendance. In 1994-1995 
a slight increase was noted in the number of students served in before and after school 
programs. 

ESS funds have allowed for an increase in summer programs from 35 districts in the 
summer of 1990 to one 174 districts in the summer of 1994. Summer programs began 
changing in two ways: (1) a broader array of programs were offered to a larger number of 
students and, (2) for the most part the summer classes became tuition free. Most districts are 
continuing to provide summer programs each year. 

Attendance data collected during the 1993-94 school term reflected a diversified ESS 
program intended to meet both the short-term and long-term needs of students. Student 
attendance patterns showed that 32% of all students were served 1-5 days and 27% were 
served 6-15 days. Approximately 29% were served 16-40 days while 11% were served more 
than 40 days. Patterns were similar in 1994-95. 



Data on the number of days 
students attended ESS programs 
throughout the state exemplify the 
diversity of programs, designed to 
meet both the short- and long-term 
needs of students. Approximately 
one-third of all students were served 
five or fewer days, while 44% were 
served 6 to 25 days and 23% of all 
students were served 26 or more 
days. 



Number of Days Attended by Students 
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Reading, math and written language have characteristically constituted the majority of 
goal areas addressed by ESS programs, although slight increases have been noted in the areas 
of science, social studies, humanities, vocational studies and practical living. Recent efforts 
have also been placed upon addressing all barriers to student learning (i.e., study skills, test- 
taking skills, portfolio development). 
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ESS Learning Goal 





Subject 1 


Subject 2 


' Subject 3 


Reading 


35 


15 


13 


Mathematics 


29 


45 


23 


Written Language 
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26 


42 1 


Science 
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Social Studies 
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10 


Arts and Humanities 
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Vocational Studies 
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Practical Living 
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* Numbers represent percentages of students 



Within the past four years, 145 innovative grants were competitively awarded to 
school districts for a total of $6,347,870 to test new ideas and methods to meet student needs. 
In recognition that ESS is a part of a school's instructional program, this strategy has 
attempted to promote systemic change within the school environment. With these grants, 
schools and districts initiate and build upon state-of-the-art instructional practices, i.e., 
Gardner's Multiple Intelligences, Deshler's Learning Strategies, and Effective Homework 
Strategies. 

Transportation continues to be a concern in ESS. Many students cannot attend ESS 
due to a lack of transportation. Where transportation is provided, the number of students in 
attendance to ESS increases dramatically. In the 1993-94 school year, 30 districts provided 
no transportation. Of the remaining districts which were using a percentage of ESS funds for 
transportation, 64 used .25% - 10% of their allocation; 48 used 10.1% - 21%; 27 districts 
used 22% - 30%; and seven districts used 31% - 43% of their ESS funds. The percentage of 
the total statewide allocation used for ESS transportation was 12.15%. 



What have been the effects of the program on stakeholders? 

Program effectiveness lor Extended School Service (ESS) programs continues to be 
evaluated in multiple ways. During the past year, data have been gathered through focus 
group meetings (AEL project), visitations to school districts, and district self-studies. Based 
upon these data sources, the following findings were reported in the Annual Report on the 
Impact of Extended School Service Programs (KDE, 1995a). 
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Student data sheets collected as part of the evaluation design indicate that the numbers 
of students who are promoted or who graduate on time have increased steadily over the years 
because of ESS. 

In 1994-95, 17,295 were noted Jo have been promoted or to have graduated on time as 
the result of having participated in ESS programs. In addition, data indicate a steady increase 
in the percentages of students who are receiving higher grades because of ESS. In 1991-92, 
41% of those students attending ESS programs 6 or more days were noted to improve one or 
more letter grades in their primary subject of focus. This number rose significantly to 67% 
improving one or more letter grades in 1993-94. In 1994-95 significant changes were again 
noted in letter grade achievement for students attending ESS programs for six or more days. 
At the time of entering their program, fewer than 6% of students had entry grades of A or B 
in the primary subject of focus, but 29% had A or B grades at the end of the school term. 
Conversely, 83% had entry grades of D or F in the primary subject of focus, while only 23% 
had D or F grades at the end of the school term. In other words, 81% of all students 
attending ESS programs six or more days were noted to improve one or more letter grades in 
their primary subject of focus while 17% maintained their prior grade. 



Entry and Exit Grade for ESS Students 



Primary Subject of Focus 




The referring teachers for ESS students also noted significant qualitative changes in 
the areas of completing homework, improved attitude toward school, and in making students 
more independent, self-regulated learners. 
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1995 Summer Term ESS Program 



Preliminary data gathered on the 1995 ESS summer program would indicate that more 
than 40,000 students were served in programs throughout the state. Further analyses will be 
available upon completion of all data entry and analysis. 



What ar e the attitudes and perceptions of stakeholders toward the program? 

In an effort to monitor the qualitative impact of ESS programs throughout the state, 
the Kentucky Department of Education has collected ongoing evaluation data through student, 
parent and teacher surveys (KDE, 1995a). Results indicate that students are very positive 
toward the program, reporting a nurturing relationship with their ESS teachers. It appears that 
ESS helps students become better learners and that the benefits extend into the regular 
classroom. Students believe that after involvement in ESS, they are more likely (1) to seek 
help when they need it and (2) to pay attention in class. Students report liking the small 
group atmosphere, more interesting teaching techniques, and the emphasis on teaching them 
how to learn. 

Parents' responses on surveys confirm that in their opinions at least, the program is 
meeting its goals. Parents report that ESS (1) helps their children feel better and more 
confident in doing their schoolwork, (2) helps students pass a subject they might otherwise 
fail, and (3 s helps them graduate on time or be promoted. 
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Regular education teachers also report very positive perceptions about the program's 
effectiveness. They believe that ESS students have (1) increased class participation, (2) 
improved problem-solving skills, and (3) improved peer relations. They also report that ESS 
students (4) more readily ask for assistance when they need help, (5) more frequently 
complete assignments, and (6) come to school more often. 

In an effort to determine the perceived effectiveness of all of the KERA initiatives, the 
Kentucky Institute for Education Research contracted with Wilkerson & Associates (1995) to 
conduct a statewide survey of 
principals, teachers, school 
council parents, public school 
parents and the general 
public. Survey participants 
were asked to indicate their 
agreement with whether or 
not the various initiatives 
were "Working Well", 

"Working Poorly", or 
"Undecided." The actual 
percentage of participants 
who felt ESS programs were 
working well are as follows: 

o ao <o eo eo loo 



Ratings of ESS Effectiveness 

*Wortcng W*tr 

] □ 



Teachers - 



Principals - 



General Public - 



Public School Parents - 



School Council Parents - 



Principals ranked ESS as the most effective KERA program with 88.6% agreeing that 
ESS programs work well. Teachers also felt that ESS programs were effective with an 
overall ranking of 2 (82.3% agreeing that ESS programs work well). School council parents 
demonstrated the lowest rating of ESS programs ranking them fourth among initiatives 
(78.1% felt ESS programs work well). Public school parents and the general public were 
both in agreement ranking ESS programs third among all the initiatives in their perceived 
effectiveness with percentages of 64.6 and 64.2, respectively. 

These data cure consistent with prior identified concerns (KDE, 1995b) where it was 
felt that the general public is no longer knowledgeable of KERA initiatives and has not been 
consistently updated by schools, districts and the state on the impact of programs. As part of 
the survey, participants were asked to identify the greatest contributing factors to a lack of 
program effectiveness. Although the majority of participants identified ESS programs as 
"working well", 35.9% of the participants who felt that ESS programs were not working well 
indicated that poor design was the most significant contributing cause. 

Additional information regarding strengths and weaknesses of the ESS program was 
obtained in a study by the KDE consultants for the ESS program. The eight regional and 
Frankfort-based consultants conducted on-site school visitations and ESS classroom 
observations during the 1994-95 school term. During these visits, interviews and surveys 
were conducted with school personnel, students and parents. In addition, school districts 
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receiving innovative grants were required to conduct a self-study of their district's needs in 
relation to the ESS program. These self-studies included observations of programs and 
surveys of parents, students, staff members and ESS teachers. 

The following are typical responses from regular teachers to the question: In my 
impression, the ESS program could have more impact in the school/district if: 

• Teachers referred uie most needy students. 

• Reguia ;hers had time to confer with ESS teachers about student goals and 
strategies. 

• ESS teachers used a greater variety of teaching strategies and less seat work. 

• Selection of competent and interested teachers was a higher priority. 

• ESS started earlier in the fall semester. 

• ESS classrooms were managed in a more structured manner. 

• All teachers were fully involved in the design of the program. 

• Parents were more involved in getting students to attend. 

Many ESS teachers have responded to a similar question: What would you need to 
have a model which you think is most effective in helping students to meet important learning 

goals? 



• Smaller number of students; 

• A better site with more space and resources; 

• A more flexible schedule and transportation for students; 

• Assistance in setting up a structure which would allow students to make optimal use 
of the time; 

• Instructional techniques for working with diverse learners; 

• More parent involvement; 

• More information from the student's regular classroom teacher about his/her 
instructional needs including past successes; and 

• Relevant instructional software. 

Parents often responded about the ESS program in the following fashion: What would 
you want your child to achieve from attending ESS? 

• Complete their homework; 

• Study for tests; 

• Bring up their grade in a subject area; 

• Be successful in school; and 

• Receive credit in some failed class (high school). 

Several areas of concern were identified drawn from general observations by ESS staff 
and from quality assurance visits. 
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1. ESS programs often run in isolation, not as an integral part of the total school program. 
School faculties seem to be unaware of the variety of ways that ESS programs can be . 
used to meet the unmet needs of students. 

2. Parents continue to lack a basic understanding and knowledge of ESS. When KERA was 
first implemented, a barrage of pamphlets and informational literature told parents about 
all programs. Parents have not been updated and provided with new information about 
ESS programs in the schools and the state. 

3. Schools do not consistently engage in strategies that lead to best programs for students. 
There is a greater need for entry and exit guidelines, individual goal setting, and strategies 
for early intervention. 

4. Much planning is done at the district level with a failure to involve schools and teachers 
in both planning and program revision. More collaboration is needed among all 
stakeholders. 

Appalachian Educational Laboratory Project 

The Appalachian Educational Laboratory (AEL), in conjunction with the Department of 
Education’s Extended School Learning Branch piloted the development of a school-level 
needs assessment and planning process. The resulting document, "Taking Stock," (1995) 
provides a process to help individual schools assess their needs to determine how to most 
effectively use ESS resources. The process was refined and piloted at Yellow Creek 
Elementary School in the Bell County School System. AEL staff worked with the principal, 
teachers' instructional aides and parents from Yellow Creek Elementary to complete the needs 
assessment process. The three areas of need identified by the process included providing 
more time and funding for individualized instruction, attempting to ensure that opportunities 
are provided in the areas of arts and humanities, and identifying positive ways to include 
parents in classroom planning. 

In addition to the development of the training document, AEL working with the Extended 
School Learning Branch developed a database of effective practices and model programs 
which can be used by elementary, middle and high school staff to produce ESS programs 
designed to better meet the needs of their individual students. Current plans are to make the 
database available through the Department of Education’s Web Server sometime during the 
1995-96 school year. 

A variety of studies meeds to be conducted to determine the impact of ESS programs on 
student learning and the ability of ESS programs to meet the needs of at-risk students. 
Although KIRIS was not designed as a measure of individual student performance, it would 
appear that KIRIS results might provide insight into the impact of ESS programs on student 
achievement. A research design that incorporates longitudinal data might investigate the 
impact of ESS programs on achievement and other indices (i.e., attendance, motivation, self- 
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esteem). Efforts also need to be made to investigate how effectively ESS programs are 
identifying and serving those students who are most at-risk for failure. 



What research is in progress in this area of KERA? 

Within the Kentucky Department of Education, the Division of Assessment and 
Accountability is currently working with the Extended School Learning Branch to use KIRIS 
data to evaluate the impact of ESS programs (Drake, in progress). Continuous assessment 
results for a group of ESS students and a control group are being compared to KIRIS results 
to try to determine if a causal link exists. 



As part of the innovative grant funding process, districts receiving innovative grants 
during the 1994-95 school term were required by the Department of Education to complete a 
district self-study. Once all documents are received by the Department of Education, a 
technical report outlining the results of the self-study process will be completed. 



Which questions or issues neer further research? 

At the present time, many questions still need to be explored. 

1. Can the impact of ESS programs be determined separately from other ongoing reform 
efforts? 



2. What is the impact of ESS programs as measured by the state's current assessment 
system? 

3. Are there alternative models other than ESS programs which would be more suitable to 
meet the needs of at-risk students? 

4. Are current ESS and other school programs identifying and addressing those factors which 
put students at-risk for failure? 
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EDUCATION TECHNOLOGY 



Douglas C. Smith and Joan Mazur 
University of Kentucky 



What does the law or regulation mandate? 

KRS 156.666 established a Council for Education Technology to develop a five-year 
master plan for education technology. KRS 156.670 specifies the intent of the activities for 
purchasing, developing, and using technology which are included in the master plan. Those 
intentions are to: 

1. Improve learning and teaching and the ability to meet the needs of individual students 
while increasing student achievement. 

2. Improve curriculum delivery to help meet the needs for educational equity across the 
state. 

3. Improve delivery of professional development. 

4. Improve the efficiency and productivity of administrators. 

5. Encourage development by the private sector and acquisition by districts of 
technologies and applications appropriate for education. 

The master plan specifies the following technology according to KRS 156.660: 
computers, telecommunications, cable television, interactive video, film, low-power television, 
satellite communications, and microwave communications. KRS 156.670 also made the 
definition of technology provided in KRS 156.666 more comprehensive by specifying video 
and computer systems, software and hardware, multiple delivery systems for satellite, 
microwave, cable, instructional television fixed service, fiber optic, and computer connections 
products, the preparation of school buildings for technological readiness, and the development 
of staff. 



In addition to use in educational instruction and administration, KRS 156.670 specifies 
that the education technology developed under the master plan, referred to as the Kentucky 
Education Technology System (KETS), be used to facilitate communication among teachers, 
parents, students, and prospective employers of students. To that end, KRS 156.670 requires 
that the master plan include the installation of a telephone in each classroom. 

KRS 156.670 also specifies that an integrated technology based communications 
system be developed under the master plan to facilitate gathering comprehensive, current, 
accurate, and accessible information relating to management, finance, operations, instruction, 
and pupil programs. 
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To what extent has the program been implemented as intended hy KF.RA 9 



Since implementation of education technology in Kentucky public schools has required 
building an infrastructure before the technology could be used as intended by students, 
teachers, administrators, and staff, determination of the extent of implementation requires a 
preliminary review of the status of the technology infrastructure. The following are indicators 
of infrastructure progress to date: 

* The KETS Master Plan and revisions have been adopted by the State Board for 
Elementary and Secondary Education. 

The KETS Implementation Plan, which outlines schedules, project descriptions, project 
management, costs, and policy issues, has been updated and adopted by the State 
Board for Elementary and Secondary Education. 

* The KETS Architectural Standards and Technical Specifications, which specifies 
standards for both hardware and software, has been updated and adopted by the State 
Board for Elementary and Secondary Education. 

* The KETS Blueprint and Selection Guide, which establishes criteria for procuring 
hardware and software and outlines planning procedures, has been developed, updated, 
and distributed to schools. 

* The KETS Building Wiring Standards , which provides guidelines for the wiring of 
schools to enhance connectivity, has been developed, updated, and distributed to 
schools. 

* The KETS Planning Workbook , designed to assist local technology coordinators in 
preparation of the district master technology plan, which must be approved by the 
Kentucky Department of Education before offers of assistance are made, has been 
developed and distributed to schools. 

* Technical Assistance Teams (TATs) have visited school sites to assist in the planning 
and implementation of KETS. 

* School districts have submitted district technology plans as prescribed in the KETS 
Planning Workbook . From those plans, inventories of school district education 
technology that met KETS standards were prepared. Each district’s "unmet need" was 
determined by comparing that district inventory to the district technology plan. A 
formula for "unmet need" was applied to determine annual "offers of assistance" from 
the Kentucky Department of Education from which school districts have purchased 
educational technology and conducted technology related staff development. 

* KETS has been implemented in a pilot site and in an innovation center in each 
regional service center area. 
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• A Student Technology Leadership Program has been developed and implemented that 
encourages the identification and development of student leaders who function as peer 
mentors of technology. 

® The Kentucky Department of Education has produced and continues to update listings 
of software and hardware vendors who meet standards determined by the Department 
and who have tendered quantitv^daea'fietRd^^^ng. 

• The Kentucky InterLinjsidg Network has been expanded, including the KETS pilot site 
and innovation centers. 

• The Kentucky Department of Education has created a model KETS laboratory, referred 
to as ETAC, in Frankfort for the testing of software and hardware and for visitation by 
school personnel. 

• Total expenditures in support of KETS by both state and local technology through the 
reporting period ending August 18, 1995, is $136,700,000. 

• Twenty-Five thousand student workstations have been installed in Kentucky public 
schools. 

Local area networks and office and communication software are installed in central 
office sites in all 176 school districts and all districts are connected to the state 
network. 

A report by the General Accounting Office (Geiger, 1995) indicates that Kentucky at 
10.2 students per computer tied for seventh in the country in computers per student. In the 
study of 7,800 schools, Kentucky was ranked seventh in the percentage of schools reporting 
sufficient computer networks, fiber optic cables, televisions, cable television and VCRs 
although a third of Kentucky schools indicated that they did not have sufficient computer 
networks. Kentucky ranked as average in modems and phone lines despite more than half of 
Kentucky schools indicating lack of this equipment. Kentucky ranked fifteenth in access to 
printers, while 20 percent of Kentucky schools indicated that they had an inadequate number 
of printers. 

A draft report, Kentucky Education Technology System (KETS) Update, prepared by 
the Office of Education Accountability (Sanders, 1995), however, concludes that the unmet 
need of teachers and students can never be resolved with the current funding level. 

According to this draft report, $280,000,000 would have had to be spent at this point, rather 
than the actual $136,700,000, to have remained on target. In addition, the KETS plan, 
without revision, will require the purchase of 42,500 more student workstations while many 
of the actually 25,000 purchased workstations will need to be replace to be consistent with 
federal depreciation guidelines. The draft report concludes that the Master Plan goal of a 
ratio of 1:6 for workstations to students cannot be met given the funding history of the KETS 
project. 



What has been the effect of the program on stakeholders? 



Research on the impact of KETS on schools in Kentucky is just now beginning to 
emerge. As several reports in this review indicate, a sufficiently robust technological 
infrastructure has been installed to make research on its impact on teaching and learning 
meaningful. 



A great deal of the initial KETS research concerned the design of the infrastructure. 
That research was conducted, primarily through competitive bidding of specifications by 
vendors, to determine hardware specifications, to design workstation environments, to indicate 
planning procedures, and to determine the specifications of the wide area network which will 
serve as the "backbone" of the system. The results of that research are contained in the 
KETS Master Plan, the KETS Implementation Plan, the KETS Architectural Standards and 
Technical S pecifications , the KETS Blueprint and Selection Guide, the KETS Building 
Wiring Standards, and the KETS Planning Workbook . 

Research regarding the implementation of education from the perspective of the 
educational users of technology are just emerging: 

Coe, Kannapel, Aagaard, and Moore (1994) are conducting the Appalachian 
Educational Laboratory s longitudinal study of KERA implementation in four rural Kentucky 
School Districts. Their findings are noted in the project’s annual report. Based on their 
observations, these researchers found that the use of instructional technology did not appear to 
be widespread. Overall, fewer than 20% of all observed teachers utilized or reported utilizing 
instructional technology as a regular part of instruction. Where class sizes were relatively 
small, computers were in almost constant use on the days visited. Younger students were 
found to work primarily with a program to teach keyboarding skills, while older students 
mostly used the word processor to type portfolio entries. 

Based on observation, factors that appeared to facilitate the use of computers included 
small class sizes and non-departmentalized classrooms. Teachers at one school where grades 
5 and 6 were departmentalized reported that students used the computers more before 
departmentalization because students stayed in the same classroom, and time on the 
computers could be worked in during the day. Departmentalization, however, made for short 
class periods and students moved between rooms, causing teachers to have difficulty working 
computer time into the schedule. In smaller schools that were able to better adapt time, the 
effects of departmentalization were not as noticeable. 



Mazur, J. (1994). Researching KERA Technological Reform: Baseline Data for 
Implementing Change and Assessing Progress. The purpose of this study was to gather 
baseline data on current practices and use of technology in Kentucky classrooms as well as to 
assess the attitudes, knowledge, and needs of teachers regarding education technology. At 
this point in the early implementation stages of the KERA technology strand, it is crucial to 
gain an understanding of current practices in order to assess the instructional effectiveness of 
the large-scale technological intervention proposed by the installation of KETS. The 
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installation of the statewide computer network is conceived as a special case of user-centered 
design: a technological interface between the schools and the myriad instructional and 
informational resources that can be accessed through the use of the interface. The focus 
question in this study, as in all user-centered design approaches, is whether the interface will 
work. If so, what are the characteristics and dimensions of instructional use? If not, what is 
needed to design a leamable, useable, and effective educational interface? Data for the 
baseline study were gathered through surveys. In addition, case study information from four 
sites (one urban, one rural, one Appalachian, and the Pilot Site) was used to amplify the 
survey results. The data have been compiled and analyzed, and the final research report is 
near completion. 

The teacher attitude and skills surveys contained in the District Technology plans of 
1 1 1 districts were used as data. The survey contained questions regarding uses of technology 
for a broad range of instructional activities (from critical thinking to preparing lesson plans). 
Questions regarding teacher attitudes toward the use of computers in instruction were also 
included. Training issues, such as adequacy of training and future needs, were elicited. 
Demographic data included subject area, grade level, and level of computer expertise. 
Descriptive statistics and one-way ANOVA were used to examine trends. 

Twenty-seven percent (27%) of the teachers reported using computers in their 
instruction. Use of computers for higher order instructional activities (such as critical 
thinking, or integrating subject matter) was low at 3%. The majority of teachers reported 
using computers in instruction for purposes of lesson planning and classroom management 
activities. Thirteen percent (13%) of those reporting instructional computer use noted 
"remediation" or "drill & practice" activities. 

Teachers were overwhelmingly positive about the possibilities for computers to 
improve instruction, increase student motivation, and individualize instruction. However, 91% 
of the random sample noted that training had been inadequate or ineffective because 
computing technology was not available to them either in labs or their classroom to follow up 
on training. The need for more "time" was noted in 67% of all surveys in response to a 
questionnaire item asking for additional comments. 



To ascertain if there were differences in computer use among subject matter areas an 
analysis using demographic data as independent variables, and instructional use and attitudes 
as dependent variables, was performed. Title One teachers were found to be most positive 
about the use of technology and to have actually implemented technology most frequently in 
their instruction. This is not surprising since the Title One teachers may have had access to 
technology through previously funded programs. 

The results of this research suggest that training needs to be tailored to individual 
teacher needs, based on availability of technology and subject area. Training also needs to 
model and explore uses of technology beyond the traditional drill and practice applications. 
While the surveys note that teachers are positive about the promise of technology, interview 
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data from the four sites strongly suggest that teachers become frustrated with the lack of time 
for learning new skills and pedagogical approaches and for planning. 



Mazur also placed a question on the Kentucky Survey, a random sample phone survey 
of Kentucky residents conducted by the Center for Survey Research at the University of 
Kentucky. Kentuckians were asked if they had a computer at home; and, if so, how was it 
used?^ Twenty-five percent (25%) reported having a home computer in a sample that was 
64% small town" or "rural." Of those responding positively, word processing was used most 
frequently for home financial management. The second most frequent use of the home 
computer noted by respondents was "school work." These results were unexpected given the 
rural character of the sample, but suggest that isolation might promote, rather than inhibit 
computer use. Respondents were not asked about telecommunications. These data are 
important for the KETS intervention because the availability of home computers has been 
linked in the research to effective, higher order use of educational software and computing 
technology for teachers as well as students. Computer-based instructional materials were 
more prevalent in 1994 than had been observed in the schools in 1993. While the 
information is clearly anecdotal, it is notable as an inductive indication of increased 
instructional use of computer-based instructional material. 

Mazur, J. (1995). The implementation of the Kentucky Education Technology System. 
This research reports the findings of two related studies that were conducted through the 
University of Kentucky Institute for Education Research to determined the extent of 
implementation, explore the use of Pankratz’ Innovation Component Configuration Map data 
collection instrument, describe teachers’ and principals’ perceptions regarding technology, and 
to make recommendations for improved practice. One of the concurrent studies reported the 
results of an external evaluation of a random stratified sample of 24 Kentucky schools at the 
various levels. The other study was a self assessment of a random geographically 
representative sample of 107 Kentucky schools. 

The finding of the two studies included: 

1. A wide variation exists in the extent of implementation in the study sample. However, 
the available technology in the majority of schools was consistent with the technology 
diffusion outlined in the District’s Technology Plan. 

2. The majority of schools in both samples had acceptable representation on the school 
technology committee, had implemented the District Administrative System, and 
showed high levels of consistency between the various district plans (technology, 
professional development, curriculum). 

3. Most schools reported limited access to networking resources and telecommunications 
tools, including very low numbers of phones installed in classrooms. School-based 
Activities in Support of KETS, which includes community information and awareness, 
was inadequate for the majority of schools in both samples. 
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4. Characteristics of schools designated high implementors in both samples include a 
Technology Rich Environment, Flexible Scheduling of Training Options, a wide array 
of Available Software Applications, and Professional Development Options in 
Technology. Factors such as Consistency of Technology Plans with other Plans, 
District Office Technology, and the Composition of the Technology Committee were 
the same for both high and low implementors. 

5. Results of the interviews showed that teachers who were using technology claimed it 
affected their practice. They were less likely to use didactic methods and tended to 
use learner-centered approaches if technology was involved. However, many teachers 
were frustrated at the lack of time and training to implement new techniques that 
integrate technology. Principals in the sample whose schools had received rewards 
believed technology was a factor in increased test scores. Most principals reported 
their schools were on track with regards to implementation as stated in their 
Technology Plans. Several were struggling with obtaining the financial matches 
needed for costly telecommunications wiring and infrastructure requirements. 

The study concluded that components related to technology implementation in school 
structures are not as crucial to high implementation as are components related to classroom 
structures. As noted in number four above, several factors contributed to high implementation 
of KETS. Factors that did not contribute to high implementation included the composition 
and structure of the school technology committee, the installation of the administrative 
network and software, and the consistency among Technology, Professional Development, and 
Curriculum Plans. Interestingly, in the overall sample, most schools are performing relatively 
well in administrative or managerial aspects of KETS, areas that are not strong contributors to 
high implementation. Thus, administrative aspects of KETS seem to be necessary, but not 
sufficient requirements for high implementation. These findings suggest the need for a shift in 
focus from the managerial aspects of implementing KETS that do not directly affect the 
classroom teacher to the instructional, communicational, and professional development aspects 
of KETS that are directly applicable to teaching and learning with technology. 

Evans-Andris, M. (August, 1995). Computing and occupational rewards among 
educators in elementary schools. This paper addressed the complexities of reward acquisition 
and occupational gain relating to computer use among educators in elementary schools. 
Findings revealed that when effectively integrated into their work routines, computers offered 
educators an important instrument with which they became more competitively positioned for 
overt occupational rewards in their schools, including promotions, lateral transfers, and 
meaningful committee assignments. 

Evans-Andris, M. (1994). Barrier to Computer Integration: Micro-interactions 
between Computer Coordinators and Elementary Classroom Teachers in Elementary Schools. 
In a study of building technology coordinators in Jefferson County, the major finding was that 
once a coordinator is designated, administrators and most teachers wash their hands of 
technology integration and use. Administrators seem to be using the "checklist" approach to 
meeting this requirement rather than appointing the building coordinators as part of a plan for 





the systemic integration of technology. This problematic finding is related to a finding from 
the researcher’s (1991) dissertation study (a longitudinal study of technology use in two 
schools) that most principals exhibited low technology use. This fact appeared to be related to 
the lack of technology integration in the schools they administered. 

Evans-Andris, M. (1995). The computer-related behaviors of adult users in nine 
urban elementary schools are described. Evans-Andris observed 72 teachers in nine 
elementary schools and interviewed half of those observed. Using ethnographic methods this 
researcher described two orientations toward computing: 1) Distancing and 2) embracing. 
Distancer and embracers exhibited characteristic styles and practices that were associated with 
their use of computers in the classroom. 

Drill and Practice was found to be the prevalent computer practice. Seventy percent 
of teachers, many who were characterized as having a distancing orientation, used the 
computer as a surrogate teacher to support remediation and/or reinforcement activities. In 
contrast, teachers who were classified as "embracers" were observed incorporating computers 
as a teaching tool primarily to enhance learning units. 

Three predominant computing styles were found: Avoidance, integration and technical 
specialization. Technical specialization was often confused, initially, with integration. 

Teachers focusing on technical specialization were teaching primarily about 
computers.However, teachers with integration styles were teaching with computers. The 
distinctions between these often confused styles emphasize the need to promote integration 
rather than basic skills which may result only in technical specialization. 

Computing styles seem to be related to school structure and to rewards and recognition 
for teachers who are succeeding at integration. This impacts on the study’s findings regarding 
the central role of the teacher in implementing computing in classroom instruction. Combined 
these factors support the author’s conclusion that computing styles may have "critical 
implications for the structure of work at the elementary school level." 



What are the attitudes a nd perspectives of the various stakeholders toward the program ? 

The public apparently is not well informed about the status of implementation of 
education technology. In a sur.ey conducted to assess the public’s awareness of KERA 
(Hougland, J. G., Berger, & Evans-Andris, 1995), more than half of respondents indicated that 
they were "not at all familiar" with KETS. Survey results indicate that the public docs 
support funding for increased use of technology in schools. Support was high for use of 
technology to improve student learning and performance as well as for teacher training. 

A statewide survey to evaluate the current status of public schools in Kentucky by 
parents, the general public, school principals, and teachers was conducted by Tom Wilkerson 
and Associates (Kentucky Institute for Education Research, 1995). Random telephone surveys 
elicited information from targeted respondents in each of the state’s eight educational regions. 
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Quotas for the target samples were proportionate to the actual distributions by region, school, 
or job functioi., and ethnic representation. The objectives of the study were to obtain an 
overall rating of public schools, teachers, and administrators, to assess changes in schools 
within the last five years, and to determine the extent of support for KERA mandated changes 
and the extent to which each target group believed the mandated initiatives were working. 
Salient findings from the Statewide Education Reform Survey prepared for the Kentucky 
Institute for Education Research by Tom Wilkerson & Associates in September, 1995 , are 
summarized in the tables below. 



TABLE 1 

How Well Is the KERA Kentucky Education Technology System Working? 



Group 


N 


Working 

Well 


Working 

Poorly 


Undecided 


Principals 


214 


82.4 


12.7 


4.9 


Teachers 


609 


83.6 


9.9 


6.5 


School Council Parents 


100 


81.1 


10.1 


8.8 


Public School Parents 


124 


60.6 


31.0 


8.4 


General Public 


214 


61.2 


29.0 


9.8 



TABLE 2 

Contributing Factors Indicated by All Professional Educators 
Who Rank Kentucky Education Technology System As Working Poorly 



Contributing Factors 


Percentage of the 102 
Respondents 


Poorly Designed 


22.3 


Lack of Time for Planning & Development 


31.9 


Not Understood by Teachers and Parents 


13.3 


Lack of Training and Technical Assistance 


55.8 


All Others 


28.0 



What other questions have been addressed? 



The following studies address other research issues in education technology: 
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Bliss T. & Hovda, R. (1995). Case Studies of KERA Related Changes in Instruction 
and School Structure. A series of eight case studies, called the Common Thread Cases, has 
been developed for use in teacher preparation and professional development. The case 
construction is part of a larger project on cases, technology and reform (Bliss & Mazur, 

1995). The cases capture the struggles and accomplishments of teachers in the midst of 
reform. Using a variety of qualitative methods, researchers at the University of Kentucky and 
University of Louisville, in conjunction with classroom teachers and graduate students, 
gathered data in selected classrooms. The cases are unique because they are true story cases 
that focus on pedagogy and depict approaches consistent with national and state teaching 
standards (INTASC and the State of Kentucky). The five secondary and three elementary 
cases cover a range of subject matter areas including math, English, social studies, American 
studies, and economic education. Rich descriptions of various instructional approaches such as 
cooperative learning, the writing process, ? d flexible math grouping are depicted in the 
cases. The cases are illustrative and are specifically designed to foster discussion of issues 
portrayed in the case. The cases are under consideration by a commercial publisher at this 
time. Further information regarding the cases can be obtained by contacting Traci Bliss or 
Joan Mazur at the University of Kentucky. 

Mazur, J. (1995). Common Thread Cases: Teachers in the Midst of Reform: 
Designing the Compact Disc Interface and Navigation Tools. This project, part of a larger 
project on cases, technology and reform (Bliss and Mazur, 1995), focused on the iterative 
design and formative evaluation of an interactive interface that supports the presentation of 
the Common Thread Cases (see Bliss & Hovda above). Using a Common Thread Case in a 
compact disc (CD-R) format, teachers have multiple ways of studying real images of good 
practice and simultaneous access to tools that intensify full participation in the story. The text 
of each case is rendered as an "animated narrative" through the use of video clips, high 
quality graphics, photography, and audio narration. Key questions in the design of the 
interface include: What is the value of using the CD for good teaching/aspiration cases? How 
can the design of the program exploit the potential of interactive CD technology and multiple 
representations of content to make the case "come alive" for the user? And, most 
importantly, how can these CD-ROM features make recognizable the practices of reform that 
are currently unfamiliar, or are only vaguely familiar, to teachers embarking on the adventure 
of transforming practice? 

The interactive multimedia portrayals of the case allow the user to read the complete 
narrative, hear the commentary by the case subject, or listen to comments by other teachers 
about the case. The program also contains tools and resources that supplement the study of 
the case. Teaching notes, case exhibits such as lesson plans, bibliographic references, and 
teacher performance standards that are referenced to specific portions of the case. 
Communication tools such as an online personal notebook or electronic mail enable the user 
to annotate the case or discuss its dilemmas with other teachers in diverse geographic 
locations. Formative evaluation of the interface was conducted and feedback from users was 
incorporated into the final design of the program. The compact disc cases are under 
consideration by a commercial publisher at this time. Further information regarding the cases 
can be obtained by contacting Traci Bliss or Joan Mazur at the University of Kentucky. 
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Mazur, J. & Bliss T. (1995). CASENET: Creating conditions for conversation and 
community for teachers in the midst of reform. In the Proceedings of the First Annual 
Conference on Computer Supported Collaborative Learning . ACM: New York. Preliminary 
research of the use of CASENET (Creatin g As sociations of Experienced and Novice 
Educators through Technology), a multi-year project that uses multimedia compact disc cases 
and high-bandwidth telecommunications to create a professional development network for 
teachers in the midst of systemic reform in the state of Kentucky is reported. Six veteran 
Kentucky social studies teachers and six student teachers at the University of Kentucky 
engaged in cased-based discussions of new teaching strategies using computer based video 
conferencing (CUSeeMe). Initial findings indicate that the CASENET tools and resources 
establish conditions fo * mutually beneficial professional conversations that encourage 
participation in a professional community. These conditions were: 

1. A shared context for discussion. Using the multimedia case as a basis for common 
experience, differences in the school contexts and teaching approach of each teacher 
could be assimilated into a discussion of the specifics of the case. This made the 
experience more relevant and stimulated multiple perspectives on an issue. 

2. An emphasis on pedagogy. The true story case was believable and included specific 
issues and strategies related to teaching social studies. This was motivating and 
challenging. Teachers, it seems, want to talk about content-related pedagogy and need 
more opportunities to do so. 

3. Flexible professional development options. The CD cases were engaging and 
convenient to use as schedules permitted. This was a key benefit for teachers working 
around school scheduling. Also, the teachers were free to discuss topics of their own 
choosing either in the case discussion or during the video conference, which led to 
their ownership of the process. All participants were eager for more cases and contacts 
with other teachers. 

4. On-line contacts that were personal, safe, and proximate. The isolation inherent in 
teaching is well documented. This isolation is both physical and conceptual. The video 
personalized the contact and the electronic link brought the two site? closer to one 
another. The benefit of having several contacts to act as sounding boards and with 
whom to discuss misgivings and ideas openly (without the threat of evaluation) was 
noted by both experienced and novice teachers. 

As the participants reported, these conditions contributed to a sense of identification 
and involvement with their own profession, a consequence of on-line collaboration also 
reported by Steeples (1993). These preliminary data demonstrate how features of the CD- 
ROM cases and the desktop videoconferencing established conditions for mutually beneficial 
professional conversations. These findings are consistent with Harrington’s (1993) work that 
suggests the potential of computer conferencing to support professional discourse. They are 
encouraging because conversations that enthuse teachers about activities such as mentoring 
and reflective practice are forms of productive participation in a community of teachers faced 
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with the challenges of KERA reform. A description of the CASENET tools, including the 
Netscape Home Page, is included. 

The research on education technology is currently in the most rudimentary and 
formative stages. This is due to several factors. First, in 1993-1994 most District 
Technology Plans were approved and the equipment has been installed in the schools across 
the state. Second, training in the use of technology is beginning in earnest, now that 
computers are available to more teachers and students. Finally, both teachers and 
administrators are beginning to plan curriculum and restructuring efforts to include the 
integration of technology. Prior to having the technology, such plans would have been 
specious at best. Given these circumstances, the present research on the education technology 
initiative has been descriptive, designed to establish baselines, and has largely addressed 
attitudes, skill levels, and basic uses of computing in schools. As teachers develop strategies 
and skills for integrating educational technology into instruction, research on the results of the 
technology initiative will be appropriate. Prior to that time, too many confounding variables 
(such as lack of training) prevent the gathering of valid data on the effects of technology on 
instruction and school management. 



What research is in progress in this area of KERA? 

The following research studies are in progress: 

Blackhurst, A. E. & Drake G. A. The University of Kentucky Assistive Technology 
(UKAT) Project is a collaborative research project between the University of Kentucky and 
Fayette County Public Schools. Funded with a $900,000 grant from the Division of 
Innovation and Development of the federal Office of Special Education Programs, the goal of 
the three-year research project is to examine the impact of the use of a recently 
conceptualized unifying model of human function to guide the development and delivery of 
assistive technology services in public schools. In order to accomplish the project’s goal, a 
series of research studies will be performed under the auspices of the UKAT Project to 
answer the following nine research questions: 

1. What school policies should be established with respect to the delivery of assistive 
technology services within the context of a model of human functioning? 

2. What is the impact of a standard set of function-based screening and assessment 
protocols on the identification of assistive technology needs of students? 

3. How effective are current information systems for locating assistive technologies to 
meet the functional needs of students? 

4. What is the effect of using a functional approach to structure the development of 
Individual Educational Programs (EEPs) that include the provision of assistive 
technology services? 
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5. What is the impact of assistive technologies that have been selected within a 
functional context on the students who are using them? 

6. What functional knowledge and skills should different school professionals possess to 
enable them to deliver assistive technology services effectively? 

7. What are the effects of using group instructional modules, interactive multimedia 
self-instructional programs, and distance learning systems to provide function-based 
assistive technology training to professionals and parents who are involved with 
students who are using assistive technologies? 

8. How effective is a telecommunication system for providing information, technical 
assistance, and support for the implementation of function-base d assistive technology 
services? 

9. How can the effectiveness of a function-based assistive technology service delivery 
system be evaluated? 

Evans- Andris, M. Computer Technology Implementation and Occupational Dynamics 
Among Teachers in Rural School Districts. This qualitative study of technology 
implementation in several rural counties in Kentucky specifically focuses on how, or if, 
schools use models; that is, do they use representatives from experienced schools as 
consultants or some other sources of expertise? The study, funded by the University of 
Kentucky/University of Louisville Joint Center for the Study of Educational Policy, 
specifically asks if school districts are "re-inventing the wheel" each time technology is 
integrated into a particular district. 

Evans-Andris also will have a new' book published by Corwin Press (Sage Publishing 
Co.) that examines the computer implementation process in schools over a four-year period. 

It details the organizational process of innovation, teachers’ responses in terms of computing 
styles, and the occupational effects of computing in terms of job skills, roles, and rewards 
available to teachers. It concludes by offering a model of computing and teaching and by 
providing a set of guidelines for administrators and teachers to use in order to develop a more 
effective, integrative computer program for their schools. Publication of this book is 
scheduled for Spring 1996. 

Smith, D. Clarifying the Role of the District Technology Coordinator. The purpose of 
this study is to analyze the current practice of technology coordinators and to provide a set of 
performance outcomes that have been validated by the membership of the Kentucky 
Association of Technology Coordinators. A delphi approach is being used to gather data on 
which consensus coding is applied. The preliminary results indicate that the idiosyncratic 
nature of the job descriptions of District Technology Coordinators require that the job 
functions be structured as a series of skill sets. A performance assessment instrument could 
then be produced for each coordinator by selecting the particular skill sets that reflect the 
specific job description of that coordinator. A further finding was that although the role of 
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District Technology Coordinator was implemented prior to the role of the School Technology 
Coordinator, the job functions of the two roles are related to the point that description of one 
requires description of the other. 



Which questions or issues need further research? 

Substantial progress has been made regarding access to education technology in 
Kentucky schools, however many questions remain that focus on the integration of technology 
into school activities. Given this infrastructure, it will now be possible to conduct in-depth 
examinations of the actual use and effectiveness of technology. The current status of the* 
initiative suggest four critical areas for research: 

1. What are the most useful training models that will result in the effective integration of 
technology into the management and instructional processes of schools in both rural 
and urban areas? 

2. What are the characteristics of schools that are currently engaged in successful 
practices that support the use of technology for improved instruction, management, 
communication and access to global information resources for teachers, students, and 
the community? 

3. What support is needed for teachers and administrators that will result in high 
implementation of KETS? What support is needed for schools that are currently high 
implementors to maintain acceptable levels of implementation? 

4. What are the effects of a widespread, mandated technological intervention such as 
KETS on the culture of the school and its larger community context? 
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PROFESSIONAL DEVELOPMENT 



Patricia L. Daniel and James R. Craig 
Western Kentucky University 



What does the law or regulation mandate? 

KRS 156.095 mandates that the State Board for Elementary and Secondary' Education 
shall establish, direct, and maintain a statewide program of professional development to 
improve instruction in the public schools as a component of the Kentucky Education Reform 
Act (KERA) of 1990. The statute describes what the State Board for Elementary and 
Secondary Education would provide: (a) an annual training program for local school district 
professional development coordinators, and (b) a series of state sponsored, professional 
development programs for certified personnel throughout the Commonwealth. 

KRS 156.0951 mandates that school districts with an enrollment of less than 20,000 
students join a consortium involving two or more districts and submit a preliminary 
professional development plan to the Department of Education. During the school years - 
1991-92 through 1994-95, the consortia received the districts’ professional development funds, 
except for the allotment to school councils made pursuant to KRS 160.345, to provide a high 
quality, coordinated professional development program to the staff of the member districts. 
After July 1, 1995, a school district could withdraw from the consortium and expend its 
professional development funds on any professional development activities it chooses. 

KRS 158.070 mandates each local board of education shall use four days of the 
minimum school term for professional development and planning activities for the 
professional staff without the presence of pupils. HB 596 amended this statute in the 1992 
Regular Session to allow school districts to use up to five additional instructional days for 
professional development activities during 1992-93 and 1993-94 if the following two 
conditions were met: (a) the local boards determine the number of uays or hours to be used 
for professional development and designate them in the school calendar, and (b) a plan for the 
use of the additional days or hours was developed by each school and approved by the district 
superintendent. The plan had to consist of training selected from a list approved by the 
Kentucky Department of Education. The General Assembly approved the continuation of this 
option for using up to five days or 30 hours for professional development from the 
instructional calendar of 175 days for 1994-95 and 1995-96. 

Program review No. 92-DPDE-029 (September 24, 1992), Use of Flexible Professional 
Development Training, stated that each district must provide the option of flexible in-service, 
(Kentucky Institute for Education Research, December 1994). The 1994 Regular Session 
passed House Bill 211 which amended the School-Based Decision Making statute KRS 
160.345 so that 65 percent of all professional development funds flow to school councils. 
Regulation 704 KAR 3:035 requires each school and local district to submit annual plans for 
their professional development training programs and requires all instructional leaders to 
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participate in a statewide training program implemented under KRS 156.101 (Kentucky 
Department of Education, 1995, May). Furthermore, the six standards contained in 704 KAR 
3:03 constitute the evaluation framework for the plans: (a) there is a clear statement of the 
school or district mission, (b) there is evidence of representation of all persons affected by the 
professional development plan, (c) application of needs assessment analysis is evident, (d) 
professional development objectives are focused on the school or district mission and derived 
from the needs' assessment, (e) the professional development program and implementation 
strategies are designed to support school or district goals and objectives, and (f) the plan 
incorporates a process for evaluating professional development experiences and improving 
professional development initiatives. 



Review of Research to date. 

To what extent has the program been implemented as intended bv KERA? 

Daniel and Stallion (1995, August) investigated the extent to which schools across the 
Commonwealth are implementing School-Based Professional Development. Site visits were 
conducted at 44 randomly selected schools from each region of the state to determine the 
extent of planning, development activities, evaluation, and support for professional 
development. The results indicated that nearly all schools sampled reported they had: (a) 
developed and implemented annual school professional development plans, (b) created support 
structures for professional development, and (c) provided traditional conferences and 
workshops. Schools which were high implementors of innovative professional development 
options differed from low implementors in the use of: 

• mentoring to provide transfer of skills from teacher to teacher (about one-third 
versus almost never); 

• action research to try out new practices (about half the high implementor 
schools versus 12% of the low implementor schools); 

• professional development activities within the school day (60% of the high 
implementors versus 11% of the low implementors); 

• local school expertise to conduct professional development activities (over half 
the high implementors are using local school expertise for professional 
development while about 29% of the low implementers do so); 

• local networks for learning and implementing new strategies (about 43% of the 
high implementors do so while none of the low implementors do so); and, 

• individual professional growth plans as an integral part of the school’s annual 
professional development program (over half the high implementors have 
teachers develop individual professional growth plans that integrate school 
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goals and their professional career goals as part of a formal school process 
versus 9% of the low implementors). 

Daniel and Stallion also found that 77% of schools sampled had annual professional 
development plans created through collaboration with school staff; however, many schools 
(57%) did not have clear mission statements or a stated set of beliefs about professional 
development. In addition, professional development is reported as taking place mostly on 
professional development days with less than one school in three sampled reporting ongoing 
year-round availability of professional development options. Data reported by the Appalachia 
Educational Laboratory (October 1995) are consistent with these findings in that they found 
that teachers generally report that curriculum development training opportunities are limited 
and of mixed quality. Furthermore, the Appalachia Educational Laboratory study found that 
teachers report time for training and development is lacking and that teachers are concerned 
about professional development activities reducing their classroom instructional time. In that 
regard, Daniel and Stallion found that approximately one in four of the schools sampled had 
implemented some type of flexible scheduling to facilitate teacher collaboration in curriculum 
planning and professional development. However, they also found that evaluation data on 
professional development activities were collected but not used in about half the schools 
studied. How representative these findings are of all Kentucky public schools is not known. 

Corcoran s (1995, April) study focused on instructional reforms in mathematics and 
science education in grades K-8 in ten schools over a five-year period. His preliminary 
report indicated that KERA has impacted classroom practice through professional 
development and has provided teachers with many opportunities to learn. Teachers reported 
they were pleased with the quality of professional development they had received, specifying 
the practical value of those experiences. While most teachers claimed they were able to 
attend any good opportunity for professional development, they also reported that their 
workloads created a time problem: (1) they did not have time to plan with colleagues, (2) 
they did not like using release time to engage in professional development, and (3) they did 
not like having professional development primarily in the summer. 

An ongoing professional development project of the University of Kentucky/University 
of Louisville Joint Center for the Study of Educational Policy and Kentucky Educational 
Television addresses Corcoran’s findings (1995) as well as other issues such as the effective 
use of technology in building professional networks. Initial data (Mazur & Bliss, 1995) from 
the project, Common Thread Cases: Teachers in the Midst of Reform, suggest that true story 
cases of Kentucky teachers, rendered in a multimedia CS format, promote the development of 
focused and meaningful professional discussions. The research, conducted with teachers at 
Shelby County High School, indicates .that the cases encourage thoughtful analysis of subject 
specific teaching and can effectively constitute flexible professional development. Additional 
research (Bliss & Mazur, in press) suggests that this combination of case-based learning and 
technology holds considerable potential for mentoring relationships among teachers at vastly 
different stages in their careers but faced with similar challenges. 
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Professional development has not focussed exclusively on teachers. The need for 
professional development for principals was noted by the Principals’ Ad Hoc Work Group in 
1994 (Kentucky Department of Education, 1995, April) when they conceptualized a study to, 
among other things: (a) assess the most effective roles of the principal to determine leadership 
needs; (b) study alternative methods for training, testing, and assessing principals and for 
reshaping" of preparation programs for building level administrators; (c) pursue development 
of a "Principal Center and/or Institute" utilizing existing principal networks and cadres; and 
(d) recommend a series of integrated training for principals that would provide for advanced 
level as well as orientation level training in KERA strand issues, collaborative leadership, and 
facilitation skills. 

Additional indicators of increased emphasis on implementation of professional 
development include an increase in funding for professional development in 1995-96 to 
$23.00 (per student based on average daily attendance) from $18.00 in 1994-95 as compared 
to $1.00 in 1990-91 and the approval in 1995 of all school districts’ Annual Professional 
Development Plans by the Kentucky Department of Education. 

The 1994 A Review of Research on the Kentucky Education Reform Act (Kentucky 
Institute for Education Research, December 1994) summarized a variety of research efforts 
that had taken place until that date. For example, the annual reports of the Office of 
Educational Accountability (1993, December 1994, December) have chronicled a variety of 
professional development initiatives that reflected the initial stages of development and 
implementation of the professional development programs and activities. In addition, the A 
Review of Research on the Kentucky Education Reform Act contained a summary of 
McDiarmid’s 1994 study which provided a valuable research framework and specific 
recommendations necessary to support teacher learning and development including 
recommendations for improving professional development and Carr’s 1994 survey of teachers’ 
perceptions of the professional development they had received subsequent to the 
implementation of KERA. In both instances, the researchers noted, among other things, the 
importance of professional development to the successful execution of education reform in 
Kentucky and the necessity of sufficient time and resources to support the requisite 
professional development. Other research reported in A Review of Research on the Kentucky 
Education Reform Act (Professional development gap analysis report by the Office of 
Regional Service Centers, January 1994) pointed to the crucial role that the principal must 
play in professional development as instructional leader and the serious weaknesses of the 
lack of sufficient professional development very early in the implementation of KERA. 



What have been the effects of the program on stakeholders? 

The implementation of KERA has required teachers to perform innovative and 
complex tasks for which most have not been trained to accomplish. As noted by several 
researchers (e.g., Daniel & Stallion, 1995, August; McDiarmid, 1994, May), if education 
reform is going to occur, then professional development is going to have to assume a more 
preeminent role than it ever has before. This view is reflected in principals’, teachers’, and 
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school council members’ reported priorities for improvements in KERA programs and 
practices (Kentucky Institute Education Research, 1995a, October). As summarized in Table 
1, principals, teachers, and school council members place a very high priority on providing 
opportunities for professional development that addresses specific needs. Unfortunately, 
professional development is often viewed as an event which occurs outside the everyday 
routines of school and lacks the follow-up, practice, and reflection needed to influence actual 
classroom teaching. This, no doubt, is one of the contributors to the stress that school 
administrators, teachers, and school council parents report they are experiencing (Kentucky 
Institute Education Research, 1995a, October). Based on the data summarized in Table 2, it 
is interesting to note that, while these perceived stress levels are quite high, the self reported 
stress levels of principals, teachers, and school council parents indicate significant changes 
from one year ago (Kentucky Institute for Education Research, 1995a, October). 



Table 1. 

The 1995 Summary of the Perceptions of Principals, Teachers, 
and School Council Parents Regarding Priorities for Providing Opportunities for 
Professional Development that Address Specific Needs 



Group 


Extremely/V ery 
High 


Moderately 

High 


Not at All 
Important 


Principals 


77.7* (+13.7)** 


17.3 (-14.2) 


3.2 (-0.4) 


Teachers 


70.4 (+4.7) 


19.8 (-8.2) 


2.9 (-2.4) 


School Council Parents 


68.2 (+6.1) 


31.7 (-2.8) 


0.0 (-1.7) 



*Percentage of stakeholder group responding to a particular category. 

** Percentage change from corresponding 1994 Kentucky Institute for Education Research survey. 



Table 2. 

The 1995 Summary of the Perceptions of Principals, Teachers, and 
School Council Parents Regarding their Persona! Levels of Stress 



Group 


Extremely/Major 

Stress 


Some/Little 

Stress 


Declined 
to Answer 


Principals 


46.6* (-27.8)** 


58.5 (+56.1) 


0.0 (-23.2) 


Teachers 


53.1 (-13.0) 


46.2 (+35.7) 


0.7 (-22.8) 


School Council Parents 


19.0 (-51.3) 


81.0 (+59.2) 


0.0 (-7.9) 



*Percentage of stakeholder group responding to a particular category. 

"Percentage change from corresponding 1994 Kentucky Institute for Education Research suney. 
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Large numbers of principals, teachers, and school council parents are experiencing 
lower levels of stress than one year ago with almost everyone sampled choosing to respond, to 
the question, which was not true in 1994. No doubt, the press for time is a contributing 
factor to perceived stress. Principals, teachers, and school council parents uniformly report 
they believe that providing teachers more time for design of curriculum and instruction (a 
significant component of professional development) should be a high priority item (refer to 
Table 3). And, as Daniel and Stallion (1995, August) note, there are instances where school 
administrators and teachers are successfully implementing productive professional 
development programs that positively impact student learning. As Daniel and Stallion found, 
many teachers are spending more time involved in professional development activities than 
the required days mandated by their school/district for professional development. They also 
found that teachers are engaged in activities that develop them professionally but which are 
not formally acknowledged as "professional development" activities. (For example, taking 
university courses, observing teachers in other classrooms, attending conferences, and 
mentoring others.) In addition, evidence of capacity building within districts and schools was 
found by Daniel and Stallion in that teachers were beginning to form networks and that 
teachers who were trained were conducting training in their own schools and/or districts. 



Table 3. 

The 1995 Summary of the Perceptions of Principals, Teachers, and 
School Council Parents Regarding Priorities for Providing Teachers More Time for 

Design of Curriculum and Instruction 



Group 


Extremely/Very 

High 


Moderately 

High 


Not at All 
Important 


Principals 


19 . 1 * 


19.2 


1.1 


Teachers 


86.7 


10.3 


1.7 


School Council Parents 


62.6 


32.4 


5.0 



* Percentage of stakeholder group responding to a particular category. 



In 1995, 231 principals were surveyed (Kentucky Department of Education, 1995, 
February) regarding their five greatest needs for professional development. They were found 
to be: (a) managing integrated curriculum; (b) using technology to manage time and personal 
priorities; (c) implementing, analyzing, and evaluating KIRIS; (d) developing strategic or 
transformation plans; and (e) interpreting test results to stakeholders. The preferred 
mechanisms for the delivery of professional development experiences were: (a) regional 
principals’ leadership institutes, (b) leadership networks, (c) statewide principal's leadership 
institutes, (d) KET broadcasts for principals only, and (e) weekend retreats for principals. 
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What ar e the attitudes and perceptions of stakeholders toward the program? 



The importance of professional development for the successful implementation of 
KERA initiatives has been noted by many (e.g., Corcoran, 1995, April; Daniel & Stallion, 
1995, August; McDiarmid, 1994, May). And, while professional development is locally 
developed and implemented and not a unified "program,” the perceptions about professional 
development by many individuals are generally positive and supportive. As reported by the 
Kentucky Institute for Education Research (Kentucky Institute for Education Research, 1995a, 
October; 1995b, October; 1995c, October), school board members, superintendents, teachers, 
principals, parents, and the general public perceive professional development to be working 
well. Greater percentages of superintendents (90.5%), principals (84.9%), teachers (75.9%), 
and school council parents (82.9%) have more positive attitudes than public school parents 
(64,8%), school board members (63.4%), or the general public (60.4%). These data are 
summarized in Table 4. 



Table 4. 

The 1995 Summary of the Perceptions of Various Groups of Stakeholders 
Regarding How Well Professional Development is Working 



Group 


Working Well 


Working Poorly 


Undecided 


School Board Members* 


63.4** 


18.9 


17.8 


Superintendents 


90.5 (+0.2)*** 


3.8 (-1.9) 


5.7 (+1.7) 


Principals 


84.9 (-6.1) 


8.5 (-0.5) 


6.6 (+6.6) 


Teachers 


75.9 (+1.5) 


17.8 (-5.9) 


6.3 (+4.4) 


School Council Parents 


82.9 (+10.5) 


15.1 (-3.9) 


2.1 (-6.5) 


Public School Parents 


64.8 (+11.7) 


21.5 (+5.9) 


13.7 (-17.6) 


General Public 


60.4 (-3.5) 


20.6 (+10.3) 


19.1 (6.7) 



*Data were not collected from school board members in 1994. 

**Percentage of stakeholder group responding. 

**Percentage change from corresponding 1994 Kentucky Institute for Education Research survey. 



It is also interesting to note that the changes in the perceptions of the various groups 
from one year ago are also generally more positive. Notable exceptions are for principals and 
the general public where declines in the percentages of individuals who thought professional 
development was working well were noted. The general public sampled in 1995 also 
reflected a sharp increase in the percentage of people who perceived professional development 
working poorly. For the public school parents sampled, dramatic increases in percentages 
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were observed for both the "Working Well" and "Working Poorly" response categories with 
respect to professional development which may be reflective that professional development is 
in fact working well in some schools and/or districts but not in others. This interpretation is 
supported to some degree by the findings of Daniel and Stallion (1995, August) which noted 
that professional development is generally perceived by practitioners as better than it was pre- 
KERA in terms of both quality and the number of options available. They also reported that 
teachers and administrators are becoming more aware of the different kinds of activities that 
provide professional development. They found that teachers reported they are willing to give 
their own time for professional development if it meets their needs but resent having to attend 
activities that do not benefit them. Corcoran (1995, April) also found that teachers perceive 
that their heavy workloads constrain their opportunities to take advantage of professional 
development experiences that meet their needs and that little attention has been given to their 
needs for subject-matter knowledge. 

One sub-group of teachers, high school department chairs, experienced a major 
increase in responsibilities with KERA. This group has no clear consensus about what their 
emerging roles should be since there has been no professional development specifically for 
them. These recent findings from a University of Kentucky study (Bliss et al, 1996) show 
that high school teachers have clearly articulated needs from their department chairs which 
are not being fully met. The two most frequently mentioned areas in which secondary 
teachers look to their chairs for assistance are "improvement in teaching" and "assessment 
techniques." 

Other data (Kentucky Department Education, 1995, April) indicate that many new 
principals perceive they are overwhelmed with professional development activities and 
overburdened with mandated programs. A common belief among the new principals appears 
to be that too much time is spent in help and information gathering sessions, not enough time 
is spent attending to the needs of their particular school, and too often professional 
development activities do not always align with the real needs of the new principal. 



What research is in progress in this area of KERA? 

1 . The Prichard Committee for Academic Excellence and The Partnership for Kentucky 

School Reform 

The Prichard Committee and Partnership for Kentucky School Reform have recently 
been funded for a two-year research project to address three sets of issues: 

a. What are current beliefs about and main sources of support for professional 
development? 

b. What is required for teachers to change practice and do current opportunities 
for professional development match these needs? 
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c. What state polices, administrative actions and other influences hold greatest 
potential for changing prevailing practice? 

2. The Kentucky Institute for Education Research 

The Institute has two projected studies related to professional development for 1996. 
The Institute will begin case studies in 30-40 schools across the state that will collect 
data on all major KERA initiatives including professional development. Since the 
cases will be holistic, focusing on specific school sites, the relationship between 
professional development and other KERA initiatives will be highlighted. 

Also for 1996 the Institute has scheduled a study of the providers of professional 
development including an evaluation of the quality of professional development 
experiences as judged by school practitioners. 

3. The University of Kentucky 

A recent University of Kentucky dissertation (Settle, 1995) which is now being 
developed for publication, focused on professional development issues in implementing 
the primary school mandate of the 1990 Kentucky Education Reform Act. In a multi- 
methods study. Settle found a significant difference in teachers’ survey responses 
concerning perceptions of principals’ commitment to implementing the primary 
program. Teachers in school identified as more successful were more likely to ascribe 
"Very Strong" support for the primary program to their principals than teachers in 
schools designated as less successful. In follow-up interviews with principals, the 
qualitative data showed that principals in more successful schools described multi- 
layered and multifaceted professional development strategies. Those in less successful 
schools were more minimalistic in their approach to professional development. 

Another dissertation in progress at the University of Kentucky (Bastin, in progress) 
focuses on the relationship between teachers’ learning styles and the types of 
professional development they have experienced. 

4. Kentucky Department of Education 

The Division of Professional Development of KDE mailed out teacher perception 
surveys to teachers in 126 randomly selected schools with councils. The survey was 
completed in the Summer and Fall of 1995. The results of this survey of staff 
development needs is planned for release in early 1996. 



What are the questions, issues, or needs for further research? 

The Kentucky Department of Education (July 1994) has established and published a 
conceptual framework to guide professional development through 1996. The framework 
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clearly provides the foundation for both a short-term and a long-term research agenda for 

investigating the effectiveness and efficiency of professional development efforts throughout 

the state over the next several years. In particular, the framework states: 

• Adequate time must be provided for results-oriented professional development to foster 
teacher and administrator growth and to enhance student achievement. The relevant 
research question is: What constitutes "adequate" time both in terms of absolute 
amount and its distribution? 

• Educators benefit most from professional development when they take personal 
responsibility for their own learning. From a research perspective, the definition of 
"benefit" needs to be examined from several perspectives to determine the "truth" of 
the statement with respect to professional development as it is practiced in Kentucky. 

• Adults, including parents who work in schools, learn best when they work collectively 
to construct knowledge, to improve practice, and to reflect upon progress toward 
shared goals with parents and the community as a whole. Researchers can directly 
investigate the effectiveness of teaming and the efficacy of various cooperative models 
implemented and practiced. 

• School reform is influenced by contextual factors that must be taken into account 
when designing professional development activities. These factors include 
organizational patterns, experience with change efforts and the maturity and experience 
of the staff. The research community should directly investigate the relationship 
between contextual factors and effective professional development to establish those 
factors that are relevant. 

• Professional development must respect and build on the craft knowledge and skill that 
teachers and administrators bring with them. A very natural research question centers 
on how best to "craft" professional development within the context of school reform as 
it is being implemented in Kentucky. 

• Effective professional development includes opportunities for participants to progress 
through the stages of professional development: orientation/awareness, 
preparation/application, implementation/management, refinement and impact in a 
timely manner. Researchers should adapt the techniques and procedures for effectively 
mapping the implementation and operation of educational innovations to investigating 
the nature of the professional development environment of Kentucky and to 
establishing how best to monitor and evaluate- the professional development of 
individuals as well as to determine the quality of those experiences. 

• Structures must be developed which allow for follow-up and support of formal training 
and which develop on-site expertise for professional development and the qualitative 
and quantifiable evaluation of such. Effective training evaluation systems should be 
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researched to determine the effectiveness and efficiency of these systems in evaluating 
professional development training. 

* Schools and districts must build the capacity to be self-renewing organizations that 
focus on critical district and school-wide needs and provide for the continuous growth 
and support of the school community. Researchers should identify and examine the 
characteristics of effective self-renewing professional development systems and 
establish the means to determine their quality and their operational effectiveness and 
efficiency. 

• Licensing and certification of teachers and administrators should be tied to lifelong 
professional development in the context of school/district/higher education efforts and 
coordinated with state and national standards. The relationship between licensing and 
certification requirements and professional development standards and outcomes 
should be thoroughly researched. 

The needs for more information about professional development in Kentucky are 
great! The tremendous paradigm shift in KERA of new roles for school professionals, new 
school structures for governance and learning new teaching and learning strategies, new 
learning for expectations and new ways of assessing learning demands far more from 
professional development efforts than the state or local schools have provided in the past. 

The state legislature has taken the initiative to greatly increase the funding for professional 
development and provided additional days. The Department of Education, as described 
above, has established a new conceptual framework for guiding professional development in 
schools. What is most needed is how best to put the new resources and ideas to work for real 
professional growth and increased student learning. For those looking for ways and 
opportunities to add to this needed knowledge base, the following ideas are suggested: 

1. Studies are needed that identify and describe exemplary practices. What schools are 
high implementors on key components of professional development? What are these 
schools doing to be successful? What factors are essential to implement successful 
school-based professional development programs? 

2. Studies are needed that evaluate the effectiveness of Kentucky's “market-driven” 
statewide system of professional development. Schools, districts and school consortia 
have been given the freedom to choose and buy professional services with a minimum 
of state control. Has this process improved the quality of professional development? 
How have dollars for professional development been used? What has been the quality 
of services provided as judged by -the practitioners and recipients of the professional 
development efforts? 

3. Studies are needed about who provides professional development to schools and what 
services are available? In a number of studies and surveys in the past year 
practitioners have complained about the lack of expertise and quality professional 
development options. What is the quality of services available? What are the specific 
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areas of expertise that need more development and who should be providing these 
services? 

4. Studies are needed on time for professional development. In recent research teachers 
say that time for training, development and planning is the most needed commodity. 
Yet many of the strategies that are being tried (e.g., additional professional 
development days during the year, released time with substitutes for professional 
development and concentrated summer workshops) are not meeting their needs. Some 
schools are being successful. What are the most promising practices to address the 
major concern of time for professional development? 

5. Studies are needed on the roles of various providers of professional development. Key 
providers include the Kentucky Department of Education, regional Service Centers, 
institutions of higher education, school consortia and private vendors. In the processes 
of past research, there have been a number of issues raised about the expected and 
actual role of different provider groups. It would be helpful to study the role as 
intended by the provider and the expected role as perceived by the school 
practitioners. For example, what are the roles of Regional Service Centers, regional 
school consortia and regional state universities? Are these complementary or 
competitive in a market-driven system? Is this desirable or does there need to be a 
clarification of roles to provide an efficient system of professional development to 
schools? 

It is important that future research on professional development assess progress with 
respect to the conceptual framework established by the Department of Education and further 
described in operational terms in the Configuration Map for Professional Development 
employed in the Daniel and Stallion (1995) study. 
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CURRICULUM FRAMEWORK 



Peter N. Winograd 
University of Kentucky 



What does the law or regulation mandate? 

KRS 158.6451 states that by July 1, 1993, the State Board for Elementary and 
Secondary Education shall disseminate to local school districts and schools a model 
curriculum framework directly tied to reform goals, academic expectations, and assessment 
strategies. The framework shall provide direction to local school districts and schools as they 
develop their curriculum and identify teaching and assessment strategies, instructional 
resources of the community, a directory of model teaching sites, and alternative ways of using 
school time. 



Review of Research to date. 



The extent to which Kentucky’s schools are changing their curricula to address KERA 
has been the focus of several major studies (AEL, 1994, 1995; Corcoran, 1995; KIER, 1994, 
1995; and Matthews, 1995). The 1994 AEL study examined the kinds of changes that had 
occurred in grades 4, 5, and 8 in four rural school districts. The 1995 AFT, study was a 
statewide study that gathered data from interviews with focus groups across the state and with 
the staff of Regional Service Centers, site visits to schools described as having successful 
curriculum development experiences, and from reviewing KDE documents. The Corcoran 
(1995) study is a preliminary report on a five-year study of ten elementary and middle school 
in four districts (three rural and one suburban) that focused on instructional reforms in 
mathematics and science education in grades K-8. The Kentucky Institute for Education 
Research (KIER, 1994, 1995) has conducted statewide surveys that provide information on 
stakeholders perceptions and attitudes about all aspects of KERA including curriculum 
frameworks . Finally, the Matthews (1995) study involved observations of and interviews with 
language arts, mathematics, science, and social studies teachers from 32 schools across the 
state. The focus of this study was on how well teachers were implementing performance 
assessment in regular instructional programs. 

The results from these studies have been organized around five key research questions: 

To what extent has the program been implemented as intended by KERA? 

• The first edition of Transformations: Kentucky’s Curriculum Framework was 

developed by the Kentucky Department of Education and disseminated to local schools 
in June, 1993. A second edition of Transformations has been developed and is 
available as of September, 1995. 
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Draft forms of the Content Guidelines for assessment in mathematics, practical living, 
reading, science, social studies, vocational studies, and writing have been distributed as 
of September 1994. Content Guidelines in arts and humanities were distributed in 
January 1995. A second version of the Content Guidelines for Assessment was 
scheduled to be released during the Fall of 1995. 

High school course outlines in 15 subjects have been distributed to all public high 
schools and school districts as of July, 1994. 

A Unit of Study Development Criteria, and A Unit of Study Reviewing Guide was 
scheduled to be distributed during the Fall of 1995. 



What have been the effects of the program on stakeholders? 

• The extent to which schools have changed their curricula varies enormously both 
within and among schools ( (AEL, 1994, 1995; Corcoran, 1995; Matthews, 1995). 

Many schools are just starting the process (AEL, 1995). 

• One of the major changes is an increased emphasis on writing and the writing process 
(AEL, 1994, Corcoran, 1995). 

• Corcoran (1995) states that there has been a “dramatic” increase in the amount of 
group work, writing, problem-solving, hands-on activities, and the kinds of 
higher-order questions asked by teachers in the mathematics and science classes that 
he and his colleagues studied. The most dramatic changes have occurred in the classes 
in which math portfolios are developed for KIRIS. Traditional instruction - textbooks, 
worksheets, and lectures - still occur with some frequency in most classrooms. 

• Corcoran (1995) also reports that preparing for the mathematics portfolios in the fourth 
and eighth grades takes an enormous amount of time and energy. In addition, one of 
the unintended instructional results of the math portfolios is that teachers spend a great 
deal of time helping students work on common prompts that later appear in the 
portfolios. While there is nothing unethical about these instructional practices, it does 
appear that the high stakes assessment system is distorting the instructional use of 
portfolios (Corcoran, 1995). 

• Teachers are using anything and everything to redesign their curriculum. Without 
textbooks and clear direction, most teachers feel frustrated and unsure of their efforts 
(AEL, 1995). 

• Just over half of the fourth and Fifth grade teachers use textbooks; approximately 
three-fourths of the eighth grade teachers use textbooks (The use of worksheets was 
observed in about a third of all classrooms visited. The use of lectures was observed in 
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about one-third of the classrooms and seemed more prevalent at the eighth-grade and 
in the teaching of social studies at all levels (AEL, 1994). 

Students were working in pairs or groups in about half of the classrooms that were 
visited. Students were involved in hands-on activities in slightly less than half of the 
classrooms visited. Teachers in fourth and fifth grade are employing authentic 
literature during reading instruction at least as much as they employ basal readers 
(AEL, 1994). 

Nine out of ten teachers in Matthews (1995) study reported using oral and written 
open-ended questions on a regular basis and eight out of ten teachers reported using 
portfolios tasks within units of instruction. 

Teachers are spending more time in collaborative discussion about instruction and 
what works (Corcoran, 1995). The content and terminology of KERA, 
Transformations, KIRIS, and the Content Standards have provided teachers with 
frameworks and language for those discussions. 

Are teachers finding the Kentucky Curriculum Framework and other curriculum guides 
useful in their attempts to revise their curricula? 

The portfolios and open-ended responses portions of KIRIS seem to be the driving 
force behind the changes in instruction (AEL, 1994; Corcoran, 1995; Matthews, 

1995). 

Nearly all the teachers in the 1995 AEL study had access to the materials provided by 
KDE. Many teachers find Transformations and other materials overwhelming and not 
tied together in any way. Many teachers find the Academic Expectations unclear, or 
too general and not specific enough (AEL, 1995). 

AEL (1994) indicated that Transformations was just beginning to play a role in 
helping the schools they studied in 1993-94. These schools were just receiving 
professional development on how to use the document. 

Corcoran (1995) indicates that Transformations has had only a limited effect on math 
and science curricula. Teachers in mathematics are finding the NCTM standards a 
very useful resource in their attempts to change classroom practice. 

Matthews (1995) indicates that Transformations, KDE-Content Guidelines, and various 
KIRIS documents are being used by 70% of the teachers they surveyed. How these 
documents were being used varies by individuals. 
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What other kinds of information and assistance do teachers need in order to change 
their curricula ? 

Teachers need additional professional development in instructional approaches that will 
help all children reach the learning goals identified in KERA (AEL, 1994). 

Teachers need more time to make the changes necessary to bring their curriculum in 
alignment with KERA (Corcoran, 1995). 

Corcoran (1995) reported that the teachers in his study felt that they needed more 
training in the subject area of science. Most of the teachers in the Corcoran study felt 
that they had adequate backgrounds in mathematics. Corcoran (p. 26) concludes that 
“Generic improvements in pedagogy such as cooperative learning will not make much 
difference if the teacher lacks the content knowledge to guide students to the 
understanding of underlying concepts.” 

Teachers need more time and training to leam how to use all the materials supplied by 
KDE (AEL, 1995). 

Schools need assistance in learning how to engage in coordinated, cohesive efforts at 
changing their curriculum in order to help student achieve KERA goals (AEL, 1994). 

Teachers need more resource documents and professional development opportunities to 
leam how to design and use performance tasks and portfolios as a regular part of 
instruction (Matthews, 1995). 

Good model lessons, performance assessment tasks, and the methods for developing 
them need to be made available to every teacher (AEL, 1995; Matthews, 1995). 

Transformations should be updated to incorporate current information in various 
national standards documents. The Academic Expectations need to be further defined 
by content standards in order to provide clarity about what students are expected to 
know and be able to do (Matthews, 1995). 

Transformations and the Content Guidelines need to be presented in a user friendly 
format (AEL, 1995). 



What are the attitudes and perceptions of stakeholders toward the program? 

• The Kentucky Institute for Education Research (Wilkerson, 1994, 1995) conducted 
statewide telephone surveys of superintendents, principals, instructional coordinators, 
school counselors, parents on school councils, parents, and members of the general 
public. Findings relevant to curriculum frameworks indicated: 
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1. The 1994 survey revealed that the majority of school professionals surveyed 
felt that the Curriculum Frameworks were working moderately well. For 
example, approximately 60% of the superintendents; 65% of principals; 59% of 
coordinators/supervisors; and 54% percent of teachers felt the curriculum 
frameworks were working well. In addition, 69% of the teachers in the study 
felt that it was extremely important to provide curriculum guides for teaching 
and learning at different levels. 

2. The 1995 survey revealed that the majority of school professionals continue to 
feel positive about the Curriculum Frameworks. Sixty -one percent of the 
superintendents; 69% of the principals; 62% of the teachers; 69% of the school 
council parents; and 73% of the school board members felt the curriculum 
frameworks were working well. 

3. Those professional educators who felt that the Curriculum Frameworks were 
working poorly gave the following reasons: 43% felt the frameworks were 
poorly designed; 40% felt there was a lack of time for planning and 
development; 32% felt the frameworks were not understood by teachers and 
parents; and 37% blamed a lack of training and technical assistance. 

4. The 1995 survey revealed that helping teachers understand what students 
should know and be able to do at different grade levels; helping schools align 
their curriculum with academic expectations; and providing teachers with more 
time for the design of curriculum and instruction are identified as very high 
priorities by principals, teachers, school council parents, superintendents, and 
school board members. 

5. Ninety-three percent of respondent groups in the 1995 survey believed that the 
performance assessments in KIRIS have changed the way teachers are teaching 
and students are learning. 



What research is in progress in this area of KERA? 

1. Faye Newsome, a doctoral student at the University of Kentucky, is focusing her 
dissertation on how teachers in one region of Kentucky are using T ransformations to 
develop new curriculum that addresses the goals of KERA. The dissertation should be 
completed by spring of 1996. 

2. Professor Lorraine McDonnell of the University of California, Santa Barbara, is 
conducting a study examining the extent to which a sample of 24 Kentucky teachers 
are teaching in a manner consistent with the KERA/KIRIS goals. 

3. KIER is planning to begin five-year in-depth case studies of 40 schools across 
Kentucky to include how schools are developing curriculum. 
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Which questions or issues need further research? 

Here are some of the most pressing questions in need of further research: 

1. To what extent are schools redesigning their curricula to address KERA’s new 
Learning Goals and Academic Expectations? 

2. What additional kind of information and assistance do teachers need in order to 
change their curricula? 

3. What kinds of guidance, training, and technical assistance do schools, as a whole, need 
in order to develop and implement cohesive and schoolwide curricula? 

4. What is the role of the Curriculum Framework If KIRIS is, as research indicates, the 
driving force behind changes in classroom practice, what role do the Frameworks 
play? What can be done to increase the positive instructional effects of performance 
assessments and portfolios while, at the same time, lessen the negative effects on 
instruction of a high-stakes system of accountability? 

5. Which schools are most advanced in curriculum development and what are the most 
successful strategies for designing curriculum? 

6. What is the optimum balance between providing enough guidance for developing 
curricula, while avoiding over-prescribing what local schools must teach? 
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MULTICULTURAL EDUCATION 



Ella Simmons 
University of Louisville 

What does the law or regulation mandate? 

Specific KERA references to multicultural education are outlined in the Kentucky 
Department of Education Program Guidelines for Multicultural Education which was passed 
by the State Board for Elementary and Secondary Education, September 1992. References to 
multicultural education in House Bill 940, are as follows: 

1. KRS 156.095: Professional Development/Teaching Students of Diverse Cultures 
"...Effective awareness and sensitivity training so teachers can motivate and nurture 
students of diverse cultures...." 

2. KRS 156.410: Selection of Textbooks "...Criteria shall require that textbooks include 
the significance of the diverse contributors to society...." 

3. KRS 156.500: Appointments to Reflect Reasonable Minority Representation "... active 
minority participation at every level of implementation be continually encouraged." 

4. KRS 158.645: Student Capacities "...grounding in the arts to enable each student to 
appreciate his or her cultural and historical heritage...." 

5. KRS 158.6451: Council on School Performance Standards: Development of Goals for 
Commonwealth’s Schools; Model Curriculum Framework (l)(b)"... Schools shall 
develop their student’s ability to: 

• become self-sufficient individuals; 

• become responsible members of family, work group, or community, including 
demonstrating effectiveness in community service, including maintaining a 
multicultural or world view; maintaining an open mind to alternative 
perspectives; displaying responsiveness and caring; recognizing rights and 
responsibilities...." 

6. KRS 160.380: School Employees - Restrictions on Appointment search to locate 
minority teachers to be considered for positions. 

7. KRS 161.028: Educational Professional Standards Board, Authority to Promulgate 
Administrative Regulations (1) (d) "...study the problems of the declining pool of 
minority teachers in the Commonwealth and submit recommendations for increasing 
the number of minority teachers...." 
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In addition, Kentucky’s Learning Goals and Academic Expectations contain references 
to cultural diversity considerations and multicultural education. 



Review of Research to date. 

To what extent has the program been implemented as intended by KERA? 

The 1994-1995 academic year saw a marked increase in multicultural education 
implementation efforts across the Commonwealth. Much of this increased activity has been 
reported by the Kentucky Department of Education Multicultural Opportunities Branch. 
According to Karen Simms (1995), the branch manager, "The Kentucky State Guidelines for 
Multicultural Education has been cited as being an exceptional model among the southern 
states within The Handbook of Research on Multicultural Education ". Simms (1995) 
characterizes this publication as "the most definitive research collection to date on 
multicultural education." 

The 1994-1995 focus of the KDE Multicultural Education Branch work has been the 
creation of heightened awareness and an orientation to strategies that support the multicultural 
education guidelines. To that end, the branch has produced and conducted a series of 
multicultural education training modules at three retreat locations in the state: (1) Bowling 
Green, (2) Frankfort, and (3) Louisville. These training retreats addressed issues and 
practices for increasing awareness of multicultural education concepts and methods, examined 
exemplary strategies, and provided an overview of technical assistance opportunities that are 
available from the KDE Multicultural Education Branch. 

In addition, the branch has engaged school district staff in a series of multicultural 
instructional model activities that were designed to assist districts in the integration of 
multicultural issues in instructional practices. In support of district efforts to infuse their 
curricula with multicultural issues and concepts, the branch provided $20,000 in Chapter II 
funds to three districts, Murray Independent Schools, Covington Independent Schools, and 
Woodford County Schools, to develop instructional models. Beyond this, the branch provided 
$21,900 in professional development funds to seven districts: Murray Independent, Covington 
Independent, Woodford County, Todd County, Hopkins County, Oldham County, and Hardin 
County Schools. Funds allowed districts to design and deliver professional development 
programs to their faculty and staff. All of these districts also received technical assistance 
from the KDE Multicultural Education Branch in support of these endeavors. 

The following is an overview of the technical assistance programs and activities 
provided by the KDE Multicultural Opportunities Branch during 1994-1995: 

July 1994 The Seventh Annual Conference for Academically Proficient African 

American High School Seniors held at Northern Kentucky University 
for 250 students 

A 
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August 1994 


Prejudice reduction workshop and sensitivity training to Fayette County 




Schools 


September 1994 


Curriculum materials development with Ashland Oil for the "Berea 
College Storv" to be published in the Multicultural Resource Handbook 


September 1994 


General assistance to the Fourth Annual Black Man’s Conference in 
Louisville 


October 1994 


Support to the Black Expo in Louisville in collaboration with 12 other 
agencies 


October 1994 


General assistance to the Kentucky Education Reform African American 
and All Children's Caucus in Louisville in collaboration with 16 other 
agencies 


October 1994 


Assistance to the Native American Mascot issues encounter in Jefferson 
County 


November 1994 


Participation in the statewide Exceptional Children’s Conference 


November 1994 


Graduate Seminar for teacher educators at Georgetown College 


November 1994 


Diversity training workshops in Bell County 


December 1994 


Assistance to counties with Title IX complaints; Title DC compliance 
workshops to Fayette and Henderson Counties with dissemination of 
Citizens for Sports Eauity Newsletter; documentation of compliance 
with Title IX in 178 school districts 


December 1994 


General assistance to the Kentucky Educational Television for 
production on girls in math and science 


January 1995 


Distribution of support materials and Multicultural Resource Handbook 
to all of the Commonwealth’s schools 


January 1995 


Materials on multicultural education and educational reform presented to 
National Association of Multicultural Education 


January 1995 


Facilitation workshops to address racial discrimination issues in two 
counties; support and general assistance to six counties, including 
support for the Jefferson County Gheens Professional Development 
Academy, Barren County Black Achievers’ Program, Christian County 
Diversity Awareness Program, Todd County Rebel Flag and Mascot 
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Sensitivity Forum, and University of Louisville Gender Equity in Music 
and Arts Education as a Career Program 



February 1995 


Co-sponsorship of the Seventh Annual Equal Educational Opportunity 




Conference 


February 1995 


Equity training for six schools; pilot of training model for summer 1995 
training efforts 


February 1995 


Publication of research on equity among African American 
administrators in the Record in Educational Leadership (Fall, 1995) 


March 1995 


Seminar and assistance for Greater Louisville Alliance of Black 
Educators; Lexington Association of Black School Educators 


March 1995 


Assistance for achieving gender equity to Kentucky Valley Educational 
Cooperative; gender equity training for three districts 


March 1995 


Downlink of teleconference, "Stop the Violence," on three occasions 


March 1995 


Assistance to the Twenty-second Annual National Black Family 
Conference 


April 1995 


Assistance with needs assessments in two counties; additional support to 
staff in county involved in racial unrest 


April 1995 


Sponsorship of Outstanding Female Athlete Seminar and Luncheon 


April 1995 


Research on alternative certification efforts in Kentucky to American 
Educational Research Association 


May 1995 


Cultural awareness training at alternative high school 


May 1995 


Overview of information of on anti-discrimination laws to cooperating 
agencies in two counties 


May 1995 


Representation of the southeast regions for the National Committee for 
School Desegregation 


June 1995 


Assistance to the South Louisville community and the Louisville Board 
of Aldermen for preventing and minimizing community unrest 


June 1995 


Assistance with the development of State School Board Mascot and 
Symbolism information package 
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What have been the effects of the program on stakeholders ? 



The KDE Multicultural Opportunities Branch has conducted on-going research on its 
implementation programs and activities. It has placed particular emphasis on the evaluation 
of its instructional program models. The final data collection phase of this study is set for 
November 1995, with findings to be released later this year in the Branch’s report. The 
primary stakeholders involved in this research are teachers and students. 

According to Simms (1995) preliminary findings from this research show that teachers 
are showing a growing awareness of multicultural education issues. In addition, teachers have 
increased their skills in providing environments and instruction that are sensitive to 
multicultural education. 



What are the attitudes and perceptions of stakeholders toward the program? 

There is little information on the attitudes of stakeholders toward the multicultural 
education component of KERA-especially attitudes of parents and the general public. 
However, some attitudinal data of a more general nature may have relevance for multicultural 
education. For example, from its July 1995 study The Kentucky Institute for Educational 
Research found in their survey on "attitudes and beliefs on which KERA was based" that 
74.5% of principals strongly agreed or agreed with the statement "All children can learn, and 
most at high levels." This figure was 22.5% higher than the findings of the 1994 survey of 
principals. In response to that same item, 19.4% of the principals surveyed reported that they 
strongly disagreed or disagreed with the statement. That is a 7.4% increase over last year. 
The number of principals who reported that they were undecided about the statement 
decreased by 29.8% from last year; 6.2% of the principals were undecided about the 
statement. 

Responses to the same statement from other groups that were surveyed in that study 
are as follows: 

Teachers: 54.9% indicated that they strongly agreed or agreed with the statement; 
39.8% strongly disagreed or disagreed with the statement; and 5.3% were undecided 
about the statement. 

School Council Parents: 73.9% indicated that they strongly agreed or agreed with the 
statement; 24.3% strongly disagreed or disagreed with the statement; and 1.8% were 
undecided about the statement. 

Public School Parents: 63.6% indicated that they strongly agreed or agreed with the 
statement; 33.4% strongly disagreed or disagreed with the statement; and 3% were 
undecided about the statement. 
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General Public: 62.7% indicated that they strongly agreed or agreed with the 
statement; 34% strongly/disagreed or disagreed with the statement; and 3.3% were 
undecided about the statement. 

The sample sizes for th^se groups were: 

N = 214 Principals 
N = 609 Teachers 
N = 100 School Council Parents 
N = 231 Public School Parents 
N = /til Members of the General Public 

/This information is valuable for future research. For example, studies might be 
performed on whether implementation of the multicultural education component of KERA 
festers increased understanding of culturally diverse groups. Students and teachers 
participating in multicultural education could be measured with attitude items such as that 
reported above. Their responses could then be compared with persons not participating in 
multicultural education, and with data from the general public. Researchers would then be 
able to estimate how much participation in multicultural programs affects key attitudes about 
the academic potential of all students. 



What other questions have been addressed? 



Several important questions related to implementation of the regulation have been 
addressed to some degree. These will be reported by KDE Multicultural Opportunities 
Branch in its report to be issued, in late 1995. In addition, some of these were to be 
addressed in a November 1995 report of the Task Force on Multicultural Education, a 65- 
person group consisting of professors, teachers, and researchers. 

1 . What conditions, knowledge, resources, and support are needed for teachers to 
implement KERA imperatives for the infusion of broad-based multicultural issues, 
concepts, and understandings into their instructional programs? 

2. How will teacher preparation programs change to better prepare beginning teachers 
and provide professional development for experienced teachers to increase their 
knowledge, understanding, and abilities for multicultural education? 

3. How will the members of school-based decision making councils acquire the 
knowledge necessary- for involvement in policy development for multicultural 
education at the local school level? 
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What research is in progress in this area of KERA? 

1. The University of Louisville, Western Kentucky University, and the Kentucky 
Department of Education, along with the Bowling Green, Owensboro, and Russellville 
Independent Schools, and the Christian, Henderson, Jefferson, Simpson, and Warren 
County Schools have been awarded a $1,560,000 three-year federal grant to study and 
increase minority teacher representation in the public schools. 

2. The Jefferson County Public Schools has implemented an alternative certification 

program, ACES, with the Kentucky Department of Education to recruit, prepare, and 
employ minorities in elementary, middle, and high schools. The district is amassing 
research data from the project. £ 

3. The University of Louisville and the Jefferson County Public Schools have formed a 
partnership for a field-based teacher education program for preparing African- 
American paraprofessionals for certification and employment in early elementary and 
exceptional child education. The project, Career Opportunities in Special Education, is 
funded for $355,000 for four years by the United States Department of Education 
Division for the Preparation of Special Education personnel and will yield research 
data. 

4. The University of Kentucky College of Education has established a research and 
professional development entity. The Center for the Study of Academic Achievement 
in Learning Environments (SAALE), that has as its research agenda: 

a. Examination of Issues of Multiple Abilities 

b. Investigations of Collaboration Learning 

c. Studies for Elimination of Educational Status Hierarchies 

d. Studies in Curriculum Development 

Three of the center s programs focus on (a) providing a research forum for scholars 
interested in studying the effects of environment on learning, (b) building the capacity 
within schools and institutions of higher education to create equitable learning 
opportunities within classrooms, (c) preparing pre-service teachers, classrooms 
teachers, and administrators who will value academic and cultural student diversity and 
who will promote student achievement through the KERA approach to instruction. 

5. The University of Louisville has implemented a $600,000 four-year project sponsored 
by the DeWitt Wallace- Reader’s Digest Fund Pathways-to Teaching Program to study 
teacher preparation for urban schools and provide an accelerated field-based teacher 
education program. The project was designed in collaboration with the Jefferson 
County Public Schools to prepare paraprofessionals, particularly minorities, for early 
and middle school certification. This project will yield research data. 
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6. Eastern Kentucky University has launched a four-year longitudinal study of teacher 
education student perceptions of cultural diversity. 

7. The Task Force on Multicultural Education, a diverse 65 member group representing 
the eight service center regions, launched its examination of concerns about 
multicultural education with the Commonwealth in March 1995. Findings from the 
group’s research in its three focus areas: (1) preservice, (2) professional development, 
and (3) community awareness and support were to be released in November, 1995. 

8. The HUES (Humans United for Educational Excellence and Equity in Schools and 
Society) Center for Multicultural Education, Research, and Professional Development 
has been established at the University of Kentucky to encourage research efforts in 
multicultural education among the faculty and graduate students at UK. Another 
initiative of the HUES Center is the publication of an ongoing monograph series that 
explores vital issues in multicultural education. The theme of the first volume is 
"Multiculturalism and Reform: Theoretical Issues and Practical Applications." It is to 
be released in the winter of 1995. 

9. Jefferson Community College faculty have conducted research on multicultural 
education to achieve gender equity in the community college setting. 

10. Morehead State University faculty are involved in several projects designed to enhance 
campus-wide awareness of multicultural and diversity issues. One of their projects 
was created to support and study faculty efforts for infusing a multicultural emphasis 
in courses through institutes and conferences. Another of its projects is designed to 
address and research gender equity issues through a new women’s studies minor and 
annual symposium for the advancement of women. Results from this study will be 
completed in November 1995. MSU’s Center for Critical Thinking has developed 
inventories that survey the degree to which students possess dispositions or habits of 
mind connected with skillful thinking and problem solving. These dispositions include: 
(1) respect for other viewpoints, (2) flexibility, (3) intellectual honesty, (4) objectivity, 
(5) open-mindedness, and (6) fair-mindedness. All have implications for multicultural 
education. 



What questions or issues need further research? 

Multicultural education has many dimensions. It is appropriate that both quantitative 
and qualitative investigations be used to study this area. Specific research methods could 
include action research, case studies, surveys of stakeholders, ethnographic studies, and 
analyses of quantitative data. Among questions that could be addressed are these: 

1. What is the direct and indirect impact of multicultural education on student learning? 





2. To what degree and in what ways will KERA’s multicultural education component 
impact other KERA components? 

3. What are the most appropriate and effective assessment strategies and evaluation 
measures— quantitative and qualitative, objective and performance-based— for use in 
research and accountability studies on KERA’s multicultural education component? 

4. What have teachers learned who have attended multicultural education training 
sessions sponsored by KDE? How much has participation in training affected 
classroom instruction? 
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KERA FAMILY RESOURCE AND YOUTH SERVICES CENTER 



Stephan M. Wilson and Phillip W. Roeder 
University of Kentucky 



What does the law or regulation mandate? 

The Kentucky Education Reform Act (KERA) was based on assumptions that: 1) all 
children should begin their school careers ready to learn, 2) parent and family involvement 
are essential for the continuity and reinforcement necessary for success in school, and 3) the 
local community must provide services to support and complement the educational potential 
of families and schools to assist child learning (Coe, Kannapel, & Lutz, 1991; Legislative 
Research Commission, 1990; Roeder 1992a). KERA builds on these assumptions by creating 
a network of Family Resource/Youth Services Centers (FRYSCs) in schools across the 
Commonwealth (Kentucky Revised Statutes, 1992). 

A sixteen member Interagency Task Force with representatives from agencies or 
groups including parents, youth, teachers, school administrators, health and other service 
providing agencies, and various Cabinet Departments was appointed by the Governor and 
assigned responsibility for formulating a five-year plan to implement the centers component. 
The plan includes a process to award grants to school districts for initiation and operation of 
centers, and a system for monitoring compliance and performance. The mission statement of 
the Interagency Task Force is “To promote the flow of resources and support to families in 
ways to strengthen their functioning and enhance the growth and development of individual 
members and the family unit.” (Interagency Task Force, 1991, p. 13) 

Family Resource Centers are to serve elementary school children and families, and 
Youth Services Centers are for middle and high school students and families. Funded centers 
are to be located in or near each school in which 20% or more of students are eligible for 
free or subsidized school meals. Family Resource Centers must address at least the following 
components: assistance with full-time child care for children ages two and three, assistance 
with after-school child care for children 4-12, health and education services for new and 
expectant parents, support and training for child day care providers, health services or referral 
to health services, and education to enhance parenting skills and education for preschool 
children and their parents. Youth Services Centers must address at least the following 
components: health services or referral to health services, referral to social services, 
employment counseling, training and placement for youth, summer and part-time job 
development for youth, substance abuse services or referral to such services, and family crisis 
and mental health counseling or referral (Interagency Task Force, 1991, pp! 13-14). 

Beyond mandated core services and referrals, FRYSCs are encouraged to develop 
relevant additional programs. These may include such unspecified areas as recreation, housing 
assistance, and crisis intervention. Overall, FRYSCs are intended to emphasize preventive 
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services and community collaboration (Family Resource Coalition, 1992; Interagency Task 
Force, 1991). 



Review of Research to date. 

To what extent has the program been implemented as intended by KERA? 

A brief summary of the budgetary and fiscal component of implementation provides 
initial evidence of implementation progress. One key implementation decision was to have 
school districts compete for funds to initiate and staff the centers. Of Kentucky’s 1,400 
schools, the more than 1,100 eligible for funding were not mandated to participate in the first 
year of operation; instead, centers have been brought on-line in stages. As an example of the 
staged process, in the first year of funding (FY 92), $9.5 million was appropriated for 133 
centers serving 232 schools (Status Report, 1991). The awards in the first year of operation 
ranged from $10,800 to $90,000 and averaged $68,100. The amounts awarded are based on a 
formula allocation of $200/year per student eligible for the subsidized lunch program in the 
school up to a maximum of $90,000. 

The second year of funding (FY 93) totaled $15.5 million for 222 centers serving 414 
schools in 103 of Kentucky’s 176 school districts (Kentucky Department of Education, 1993). 
The range of awards continued to be from $10,000 to $90,000 with almost half the centers 
(108) receiving awards of $80,000 to $90,000. 

In FY94, 158 centers were added (i.e., 380 total), while in FY95, 75 were added to 
bring the total to 455 centers serving 752 schools. The FY95 budget for the FRYSC program 
is $31,848,000 (state general funds), which was approximately double the FY93 budget. In 
FY96, there are a total of 545 Centers serving 861 schools with state general funds of 
$37,276,000. After the last legislative session (1994) which extended the Interagency Task 
Force until December of 1997, the State Implementation Plan was revised. Assuming 
adequate funding, the plan calls for the addition of new centers for each of the next two 
years. 



As the program enters the fifth year of awards and center operations, the basic policy 
implementation question continues to be whether the Interagency Task Force, local centers, 
and advisory councils are carrying out the legislative mandate and spending public funds as 
intended by the designers of the legislation. Two early studies of implementation (AEL, 1991; 
Roeder, 1993) concluded that the policy was being implemented quickly but effectively. 

Based on site visits -and interviews with a sample of center coordinators, teachers, 
administrators, parents, and various state-level officials, the studies found that the Family 
Resource and Youth Services Centers program within KERA was a well-designed component 
of the total education reform package. A later report by Kentucky Youth Advocates (1994) 
examined 17 centers in Jefferson County and concluded that the Family Resource Centers 
(FRCs) are working as intended, the center coordinators are able, and the centers are highly 
accountable, but parent participation in center activities varies. 
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Rose and Shepard (1994) interviewed a sample of FRYSC coordinators and concluded 
that beliefs about parent inadequacies often caused coordinators to react in an authoritarian 
manner and attempt to solve at-risk families’ problems for them rather than to help them to 
identify their own problems and provide tools which will enable families to better manage 
their lives. The researchers argue that FRYSC coordinators must move from increasing 
involvement by parents and teachers, to monitoring, then to participation, and finally to 
empowerment. 

A researcher at Vanderbilt University engaged in a three-year qualitative evaluation of 
the FRYSC program (Smrekar, 1994) found that in case-studies of an urban and rural center, 
the conflict and confusion over the boundaries between the “private zone of family life and 
public responsibility of education” were somewhat mediated by the family resource centers. 
The attempt to balance the need to “separate” from a school building that some parents see as 
threatening or intimidating combined with the need to work closely with school staff led to 
contradictory messages of “inclusion” to parents and “exclusion” to school staff. Smrekar 
(1994, pp. 22-23) suggests the need to re-examine existing family-school interactions, 
especially the role of teachers, in any attempts to develop programs to link schools and social 
services. 

The FRYSC Branch in the Cabinet for Human Resources (CHR) has sponsored two 
qualitative evaluations of the FRCs (Cannon, Kalafat, & Illback, 1994; Kalafat & Illback, 
1993). The first report was based on site visits and interviews at ten centers, all of which 
were “fully functioning” and “had become vital elements in their school-community within a 
relatively short period of time.” The report concludes further that the ten centers in the 
assessment were “unquestionably excellent,” and there is “considerable evidence regarding 
their effectiveness and cost-efficiency.” (p.3). The second report is based on visits to 
additional centers. These led Cannon et al. (1994) to conclude, “...these 22 centers are 
‘highly cost efficient, school-based, family oriented, multi-service programs.'” (p. 20). In 
addition, the needs-based community programming is supported by families and the majority 
of school and community personnel. 



What have been the effects of the program on stakeholders? 

A review by the Office of Education Accountability (OEA, 1994) concluded that 
“FRYSC’ s have enabled education, social services and health-care providers to reduce or 
eliminate the barriers to learning which confront many of our children.” (p. 164). However, 
the report suggests the need for more guidance on fiscal issues and administrative policies. 
The OEA report makes a -series of- recommendations concerning financial management and 
administrative practices. 

In addition to the qualitative studies mentioned previously, there is one large-scale 
project that has collected quantitative data on center programs. The Family Resource Center 
in the CHR sponsors an ongoing quantitative evaluation project (Illback, 1993: Illback & 
Kalafat, 1995) in which program and participant data are collected from FYRSCs using a 
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computerized management information system. The 1995 report is based on data collected 
from 370 centers including 25,555 families and 31,196 students. The report focuses on who is 
being served, what problems clients present, what services have been accessed or provided, 
and what outcomes are associated with program involvement. 

It is impossible to summarize adequately the findings of this long-term research 
project in this brief report, but there are several conclusions to note as follows: the program 
appears to be serving the intended population; no single problem predominates and 
participants experience several problems that probably interact; FRYSCs do not serve a 
clinical population, but rather a diverse group of at-risk youth and their families with complex 
and broad-based needs; centers provide and/or broker a variety of programs that are diverse 
and flexible; and there are continuing unmet needs for many families (Illback & Kalafat, 

1995: 8-9). 

Preliminary outcome data on a subsample of children and families suggest some 
improvements in classroom performance variables as rated by teachers, but “more global 
measures of change (e.g., grades, achievement) do not register substantial gains (not 
surprisingly, given their long-term nature).” (Illback, 1993: 5) The author recommends the 
FRYSC program “receive full and unqqalified support as a unique approach to supporting 
children, families, and the schooling process, and a vital component of school reform.” 

Preliminary data from a study conducted by the Kentucky General Assembly Office of 
Education Accountability (OEA) confirm that the presence of FRC coupled with School- 
Based Decision Making in ten randomly selected elementary schools made a positive 
difference in Kentucky Instructional Results Information System (KIRIS) assessment results. 
The experimental group out-performed the control group both in total point gain and 
percentage growth from baseline over a two-year period. These results held for the 
Transitional tests and for Writing Portfolios (Terry, 1995). 

What are the attitudes and perceptions of stakeholders toward the program? 

Surveys conducted for the Kentucky Institute for Education Research (Tom Wilkerson 
and Associates, 1995) provide data from several groups of stakeholders on perceptions of how 
well the FRYSC program is working. School principals and teachers view the program as 
quite successful. Seventy-seven percent of principals surveyed state that the program is 
working well with only 6% stating that it is working poorly and 17% undecided. Teachers are 
only slightly less favorable with a margin of 72 to 7% perceiving the program as working 
well. The proportions for school -council parents -is virtually identical to teachers (71-7%). 
However, the perceptions of parents with children in public schools and the general public are 
much less positive than other groups of stakeholders. Public school parents feel the FRYSC 
program is working well versus working poorly by a margin of only 51-32%, while the 
margin for the general public is 47-32%. These data suggest that parents and the public may 
not be aware of the relative success of the FRYSC program as suggested by the previously 
reported implementation studies. 
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What research is in progress in this area of KERA? 

Several of the studies summarized above are based on continuing research or 
evaluation projects. The qualitative analyses of local centers by Kalafat and Illback (1993) 
and Cannon et al. (1994) and the quantitative assessments by Illback (1993) and Illback and 
Kalafat (1995) are part of the overall FRYSC evaluation plan as approved by the Interagency 
Task Force and sponsored by the CHR. The research being conducted by Smrekar (1994) is 
part of an ongoing project sponsored by the Danforth Foundation. 



Which questions or issues need further research? 

In addition to the question of how successfully the policy is being implemented, there 
are two policy impact questions that need further research. The first policy impact question is 
the degree to which center programs improve the well-being of children and families? The 
second policy impact question is whether improved well-being of children and families leads 
to improved student academic performance? The study by the Appalachian Educational 
Laboratory (AEL, 1991, p.l) suggests that the few centers evaluated in the first year of 
operations “have already affected the lives of several students and families in ways that seem 
likely to result in improved student attendance and performance,” but that conclusion is based 
on a small, non-random sample of centers and students very early in the implementation 
process. 

Despite early success in the first stages of implementation, there are likely obstacles to 
assessing program outcomes. Several important questions relate to program evaluation and 
differences between implementation success and outcome success. Although early studies 
suggest implementation is successful, evaluating and assessing outcomes of the program 
present difficult problems (Russo & Lindle, 1993; Roeder, 1992b). Even if the policy is 
implemented successfully, will the lives of families and children be made better? What 
measures and data will be collected to answer this question? Will students learn more or be 
more successful in school as a direct result of center programs or services? To what degree 
can any changes be attributed to center programs? How many years of operation are 
reasonable to expect that center programs might have positive and measurable impacts in 
these areas? These questions pose difficult issues of research design including measurement of 
student and family well-being, selection of samples of centers and students, and controls for 
other intervening factors. Even if data show positive relationships between center programs 
and improved student and family well-being, to what degree can other plausible explanations 
for improvements in student/family functioning be identified and controlled? 

With the exception of the ongoing study commissioned by the Family Resource Center 
in the CHR (Illback, 1993; Illback & Kalafat, 1995), systematic data are not yet available to 
answer outcome questions. Since it is not likely that center programs (as well as the entire 
KERA package) will have immediate, measurable impacts, it is important to gather data and 
conduct evaluation research over time. To assure accountability, statewide data-gathering and 
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evaluation is imperative. At the same time, local flexibility and unique programming must be 
recognized and preserved. 
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PERCEPTIONS AND BELIEFS ABOUT EDUCATION REFORM 

IN KENTUCKY 



Roger Pankratz 

Kentucky Institute for Education Research 



The Role of Surveys and Public Opinion Polls in Research About KERA. 

One of the guiding principles for the work of the Kentucky Institute for Education 
Research states: 

KIER should support an agenda of multi-faceted research that includes case 

studies of best practices; longitudinal studies of program, practices, schools and 

school systems; in-depth studies of key elements; and all types of surveys that 

provide useful information on the reform process. 

It is within this context that a review of surveys and opinion polls is presented as one 
source of data that adds to our knowledge about KERA. Opinions and perceptions about 
specific programs related to KERA are included in other reviews in this document. The 
discussion in this review section will address knowledge about KERA, general support for 
KERA, strengths and weaknesses of KERA, and perceived effects of KERA. 

Data from two major statewide survey efforts will be featured in this review: 

• Kentucky Surveys conducted from 1990-1995 by the University of Kentucky 
Survey Research Center (Berger, Hougland, & Kifer 1993 and Hougland, 
Berger, Evans- Andris, & Kifer 1995) 

• Education Reform Surveys conducted by Wilkerson and Associates for the 
Kentucky Institute for Education Research (KIER 1994, 1995) 

These surveys polled public opinions and those of school professionals throughout the 
state over time and sampled responses on a variety of aspects and issues of the Kentucky 
Education Reform Act of 1990, In addition to these two major sources of data information 
from the Kentucky Blue Grass Poll (1993), mail-out surveys conducted by the Kentucky 
Institute for Education Research and a recent statewide survey conducted by the Preston 
Group, Inc., (1995) of Lexington will be summarized. 

Opinion polls and surveys can be very informative if one makes the assumption that, 
for many people, perception is reality. However, the interpretation of polls and surveys needs 
to be tempered with reasonable caution. Experience with opinion polls over the last three to 
five years strongly suggests that how questions are asked, events that occur just prior to polls, 
knowledge and interests of respondents about the survey topic, greatly affect the results. 
Nevertheless, public opinion polls and surveys are an important ingredient in what we need to 
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know about school reform and how best to guide implementation to increase the benefits of 
education reform. 



Who knows about KERA? 

The 1994 and 1995 Statewide Education Reform Survey by Wilkerson (1994, 1995), 
the Kentucky Survey (Berger, Hougland, & Kifer 1993) and the Blue Grass State Poll (1993) 
all asked Kentuckians across the state about their knowledge of KERA on specific aspects of 
KERA. Table 1 presents the specific question asked and the percent who said they did not 
know or were unaware of KERA or specific KERA programs in several surveys over the past 
five years. 



Table 1. Percent of Respondents Who Are Unaware of KERA 







Percent 


Question 


Source 


Unaware* 


Have you heard or read anything about the changes that the Kentucky 


Bluegrass State Poll 




Education Reform Act (KERA) has brought to Kentucky public schools? 


1993 


56 


The Kentucky Legislature passed the Kentucky Education Reform Act in 1990. 
The law is more commonly known as KERA. This law required significant 
changes in how public schools, pre-school through grade 12, are organized, how 


Wilkerson & 




teachers teach, and how students learn. KERA mandated that all schools in the 
state make major improvements and it substantially increased the amount of 


Associates 1994 


52 


money state government spent on education. How would you rate your 
knowledge and understanding of KERA at the present time? (Includes Low and 


Wilkerson & 




Nothing at All options.) 


Associates 1995 


54 




The Kentucky Survey 


69 


Have you heard about the provisions of KERA that will eliminate grade-level 


1990 1992 


55 


designation for children in kindergarten through third grade? 


1994 


46 




The Kentucky Survey 




Some of the provisions of KERA increase the amount, cost, and type of 


Spring 1992 


68 


statewide testing, like having students write essays and do science experiments 


Fall 1992 


59 


instead of taking multiple choice exams. Have you heard about these 
provisions? 


Spring 1994 


58 


As part of the KERA school reforms, a new type of statewide student testing 
has begun. It emphasizes critical-thinking skills, like writing essays and doing 
science experiments, instead of taking multiple-choice exams. Have you heard 


Bluegrass State Poll 




or read anything about this new testing program? 


1993 


56 


Have you read or heard anything about the changes in the public school system 


Statewide Science 




as part of a major educational reform effort in Kentucky? 


and Technology 






Survey 1995 


41 



* Unaware includes Do not know , No, Low and Nothing at ALL responses depending on the question asked. 
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From these data it appears that about half of Kentuckians have little or no knowledge 
of the state’s school reform initiatives. Also, it does not appear that the proportion of 
uninformed has changed much over the past several years. 

Surprisingly, the proportion of Kentucky public school parents who say they know 
little or nothing about KERA programs is not much different from that of the general public. 
In both 1994 and 1995 as part of the Statewide Education Reform Survey, parents across the 
state were asked to rate their knowledge of KERA at the present time. In 1994, 45 percent 
and in 1995, 48 percent of parents rated their own knowledge and understanding very low 
(Wilkerson 1994, 1995). 

During a fall statewide survey of persons likely to vote in the 1995 state election 
conducted by the Preston Group, Inc. (1995), 85 percent gave an opinion about whether 
KERA had been positive or negative for Kentucky schools. However, when asked to identify 
the single greatest strength and weakness, 48 percent and 43 percent respectively offered no 
response for the single greatest strength or weakness. One-third of those responding to the 
greatest strength gave the response of “nothing.” 

Comparative data on how different groups perceive the general effects of KERA 
suggest that parents with more knowledge of KERA perceive the school reforms much more 
positively than parents or the public with less knowledge. Figure 1 shows how school council 
parents, parents not on school councils and the general public think local schools have 
changed in the past five years. 



Figure 1 



PERCEPTIONS OF HOW THE PU1UC SCHOOLS HAVE CHANCED OVER THE LAST FOUR YEARS 




School Coorvd P*r*rU PuWc School Paronta Ganarti P ubhc 

■ »•«**■ ■ won* □ Not Omtf dMuch 



208 



O 



253 



These data suggest that a critical factor to perceiving change in schools as positive is 
knowledge and involvement. While one could argue that parents and/or the public may not 
have a need to know about specific school reform initiatives, it appears that it is absolutely 
essential to inform and involve parents and the public to have sustained support for reform. 



Who Supports KERA? 

Support for Kentucky’s Six Learning Goals 

One of the earliest developments of education reform in Kentucky was the work of the 
Council on School Performance Standards. Six learning goals that defined what students 
should know and be able to do were proposed as the centerpiece for the curriculum and 
became part of the House Bill 940 legislation. The law also charged the Council on School 
Performance to define these six goals into “measurable terms.” This led to the development 
of 75 valued outcomes. These 75 outcomes, or learning targets, have come under attack by 
various individuals and groups that say the new goals and outcomes do not require students to 
learn the content they will need, and that some of the goals propose that schools teach values 
that should be left to the home. As a response to these criticisms the 1994 Kentucky General 
Assembly passed legislation preventing the state from assessing two learning goals: 

“becoming a self-sufficient individual” and “becoming a responsible member of a family 
work group or community” as part of the school accountability system. In addition, the 
Kentucky Board for Elementary and Secondary Education revised the language of the 57 
remaining outcomes and renamed them the 57 Academic Expectations which remained 
consistent with the original learning goals. 

In the spring of 1994, the Kentucky Survey (Berger, Hougland, & Kifer 1994) polled 
the general public about their support for Kentucky’s six broad learning goals. Table 2 shows 
the extent of strong support for these goals. 
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Table 2. Support for KERA Goals 1994 



Goal 


Percent 

Strong 

Support 


Students should be able to apply basic communication and mathematics skills to situations they will 
experience in life. 


78 


Students should develop their abilities to apply core concepts and principles from science, mathematics, 
social studies, arts and humanities, practical living studies and vorational studies to what they will 
encounter in life. 


71 


Students should develop their abilities to become self-sufficient individuals. 


84 


Students should develop their abilities to become responsible members of a family, work group or 
community. 


83 


Students should develop their abilities to solve problems both in school and in a variety of situations 
similar to what they will encounter in life. 


82 


Students should develop their abilities to connect and integrate knowledge from all disciplines into their 
own knowledge bases. 


73 



Only about 1 percent of respondents said they opposed these goals. 

In the 1995 Statewide Education Reform Survey (Wilkerson 1995), both parents and 
the public statewide were asked if they supported or opposed schools teaching for “self- 
sufficiency” and “responsible group membership.” Parents and the public also were asked if 
they supported or opposed schools holding students accountable for these two goals. Nearly 
90 percent of parents and the public said they supported schools teaching for these goals and 
nearly 80 percent said they supported schools bolding students accountable for these two 
goals deleted from the testing system. 



Support for KERA Beliefs 

The Legislative leaders in concert with the designers of the 1990 education reform law 
began their work with a number of basic assumptions about teaching, learning and school 
governance that guided the inclusion of several reform initiatives. 

Table 3 shows the results of the 1995 Education Reform Survey (Wilkerson 1995) for 
five major beliefs underlying KERA for .six major. stakeholder .groups. 
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Table 3 * Agreement of Six Stakeholder Groups with Beliefs Underlying KERA Surveys 

completed in the Summer of 1995 



Belief or Assumption 


Percent Who Registered Agreement With Belief or Assumption 


Principals 


Teachers 


Council 

Parents 


Parents 


Supts. 


School Bd. 
Mbrs. 


All children can learn and most at a high level. 


75 


55 


74 


64 


81 


70 


We should set high standards of achievement 
for all children. 


95 


92 


94 


91 


97 


91 


It’s not enough to require that students show 
their knowledge of facts - they must also 
demonstrate they can apply what they know in 
real life situations. 


93 


92 


98 


95 


94 


88 


Decisions related to instruction can best be 
made at the local school level. 


92 


88 


96 


82 


75 


81 


In primary school students should not be 
labeled as belonging to a specific grade level. 


44 


36 


49 


43 


59 


33 



From these data it is obvious that there is strong support for high standards and for 
requiring the application of knowledge. However, reactions to the underlying beliefs about 
the primary program and school accountability are mixed and identify several of the most 
controversial aspects of Kentucky’s school reform. 



General Support for KERA 

While there is overwhelming support for the learning goals of KERA and strong 
support for most of the beliefs underlying KERA, general approval of Kentucky’s school 
reform is less evident and appears to have declined since 1990 when the Reform Act was 
passed by the General Assembly. For the past five years the Survey Research Center at the 
University of Kentucky has polled Kentucky households by telephone regarding their 
perceptions about a variety of current issues including school reform. 

Figure 2 shows the responses of the general public over time to the following 
question: 

Would you say you strongly approve, somewhat approve, somewhat disapprove 
of the Kentucky Education Reform Act designed to change the education 
system in the state? 




211 



256 



Percent 



Figure 2 - Approval and Disapproval of the General Public 



Percent of General Public Approval and Disapproval for KERA 




j HI Approve Q Disapprove 

1 Source: The Kentucky Survey, The Survey Research Center, University of Kentucky 



In the 1995 Statewide Education Reform Survey (Wilkerson 1995, KIER 1995a, KIER 
1995b) school parents, the general public and school board members were asked the following 
questions if they indicated they had at least some knowledge of KERA: 

“Based on what you have seen, heard or read about KERA, would you say, at 
this time, you generally support or oppose the changes required by the 
Education Reform Act of 1990? Would that be very supportive, moderately 
supportive, moderately opposed or very opposed?” 

Table 4 shows the responses of the three groups surveyed in the summer of 1995. 



Table 4 - Reported Support for KERA (Percent) 



Respondent Group 


Very/Moderately 

Supportive 


Very/Moderately 

Oppose 


School Parents 


56 


27 


General Public 


53 


39 


School Board Members 


62 


26 



The total percents do not add up to 100 due to responses of “don't know" or “ undecided . 



From October 25 to November 4, 1995 researchers from the Preston Group, Inc., in 
Lexington, Kentucky, called Kentucky households in which one or more adults had voted in 
three of the last five general elections. Only respondents who reported to be registered voters 
were included in the final sample. As part of a survey to gain information on potential voters 
in the November 1995 election, the following question was asked: 

“KERA imposed sweeping changes in the structure and style of education in 
the state. In general, do you think KERA has been very positive, somewhat 
positive, somewhat negative or very negative for Kentucky schools?” 

Of 806 adult respondents 37 percent thought KERA had been very or somewhat 
positive and 48 percent thought KERA had been somewhat or very negative for Kentucky 
schools. 

While this survey shows a lower general approval rating of KERA than previous polls, 
it should be pointed out that the respondents were a select group of the general population 
and the timing of the survey was just prior to a statewide election where the effects of KERA 
were a prominent campaign issue. 



What are the Perceived Effects of KERA? 

Debates about KERA in the media often include arguments about whether KERA has 
been good or bad for Kentucky schools. The 1995 Statewide Education Survey (Wilkerson 
1995, KEER 1995a, KIER 1995b) asked different groups the following question: 

“Thinking back over the past five years or so would you say that the local 
schools in your district have mostly changed for the better, mostly changed for 
the worse, or have not changed at all?” 
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Table 5 shows the percent of respondents who think schools are better or worse since 
the enactment of KERA. 

Table 5 - How Public Schools Have Changed 
Over the Last Five Years - 1995 



Respondent Group 


Better % 


Worse % 


Superintendents 


84 


4 


Principals 


84 


3 


Teachers 


60 


19 


Council Parents 


74 


9 


School Parents 


42 


26 


General Public 


39 


23 


School Board Members 


57 


22 



Percent not shown represent the respondents who say the schools have not changed or 
are undecided or gave no response. 

In late summer 1995 a survey, conducted for the Kentucky Science and Technology 
Council (1995) by Horizon Research International polled legislators, business and community 
leaders and the general public on the question: 

"How successful or unsuccessful has Kentucky been in improving its schools 

over the past five years?" 

Based on the opinions of about 600 respondents, 60% believed the effort to improve 
schools was successful compared to 31% saying that Kentucky's school improvement effort 
has not been successful. As discussed earlier, those stakeholders most involved with schools 
and most knowledgeable about schools perceive KERA to have had a more positive effect 
than those more removed from where teaching and learning take place. 



Positive Results or Strengths vs. Negative Effects or Weaknesses 

Over the past two years there have been several surveys that have asked respondents 
to identify strengths and weaknesses or positive and negative effects of KERA. Table 6 
presents the most frequently mentioned positive aspects of KERA given by different role 
groups and Table 7 shows the negative aspects most frequently mentioned when given the 
opportunity for an open-ended response. 





Table 6 - Most Frequently Stated Positive Aspects of KERA* 
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It IS evident from Table 6 that the increased emphasis on writing, additional funding to 
schools, the Family Resource and Youth Service Centers, increased parent involvement and 
new instructional strategies top the list of positives that most groups recognize as effects of 
school reform. The most frequently stated negatives as shown in Table 7 focus on 
Kentucky’s new testing system, concerns about School-Based Decision Making and the 
rimary Program, stress on teachers and administrators and increased paperwork. The 
positives appear to be about new opportunities for learning and new services that are 
perceived to enhance learning. The negatives are about those aspects where a change in 
beliefs and behavior are required. These data support the notion that Kentuckians want 
improved learning opportunities but real change is painful. 

7 k !°° k 3t thC individual responses of the five surveys reviewed in Tables 6 and 

7 tHa t th °, S , e T rC removed from where teaching and learning take place, i.e., parents 

an the general public, have mostly general impressions about reform and few specifics they 
can address, and (2) the perceptions about positive and/or negative effects are highly 
dependent on interests and concerns of a specific role group 



Perceptions About the Effects of Miscellaneous Aspects of KERA 

The 1995 Statewide Education Survey asked respondents about their 
agreement/disagreement with popular themes that had been recorded by researchers 
conducting focus groups across the state in 1994. Table 8 shows the percent who believe 

statements frequently made by school administrators, teachers or parents about KERA are 
very or somewhat true. 



On the positive side there appears to be the perceptions from stakeholder groups that 

h n e - f ateglCS intr0duc f in have broad ened the learning experiences for students 

and that the new more performance-based assessments have changed the way teachers are 

eaching and students are learning. On the negative side, teachers and parents are concerned 
about emphasis on writing and thinking processes short-changing the content that should be 
learned. Also there is the apparent negative reaction to the state’s new school accountability 
program and the aggressiveness of avid supporters of KERA to affect change 
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Table 8 - Perceptions About KERA Retated Effects 





Percent who Believe the Statement to be Very or Somewhat 

True 


Statement About KERA Related Effects 


Principals 


Teachers 


School 

Council 

Parents 


Superin- 

tendents 


School 

Board 

Members 


A. The new learning goals and 

strategies introduced in KERA 
have made learning fun for a lot 
more children. j 


83 


76 


85 


80 


65 


B. Teachers, school administrators 

1 parents often feel too 
intimidated by colleagues or 
superiors to speak their true 
feelings about KERA. 


65 


69 


70 


65 


83 


C. The heavy emphasis on 

communication and thinking 
processes in KERA short-changes 
students on content they need to be 
successful in life. 


62 


75 


66 


53 


59 


D. The performance assessments in 

KIRIS have changed the way 
teachers are teaching and students 
are learning. 


99 


94 


94 


89 


87 


E. KERA will enable students to do 

as well or better on college 
entrance exams such as the ACT 
or SAT than a traditional 
curriculum. 


55 


43 


61 


29 


24 



Summary 

A number of public opinion polls and surveys of stakeholders have been conducted 
since the enactment of Kentucky’s school reform law. The data from these surveys have 
provided useful information about the perceptions of different role groups over time and have 
provided important information about specific aspects of KERA. The results of these polls 
and surveys have confirmed that discretion is a must in their interpretation and that it is 
usually not advisable to compare different surveys. How questions are asked, the timing of 
the survey, the population surveyed, and events all have been shown to affect the results. 
Trend data using the same survey over time with similar populations have provided useful 
data about changes in perceptions. Thus, surveys and opinion polls can provide useful 
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information in guiding education reform when used with discretion and along with data from 
other types of resaerch. 
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